Poster #20 Bayesian Nonparametric Federated Learning of Neural - PowerPoint PPT Presentation

Jan 20, 2023 •30 likes •120 views

IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 ICML 2019 Poster #20 Bayesian Nonparametric Federated Learning of Neural Networks Mikhail Yurochkin Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, Yasaman Khazaeni IBM

IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 ICML 2019 Poster #20 Bayesian Nonparametric Federated Learning of Neural Networks Mikhail Yurochkin Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, Yasaman Khazaeni IBM Research, MIT-IBM Watson AI Lab June 12th 1
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 Federated Learning
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 Model fusion perspective
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 Probabilistic Federated Neural Matching
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 Simulated heterogeneous Federated Learning on MNIST Client 1 Client 2
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 Examples of first layer weights Neuron 12 Neuron 21 Neuron 49 Client 1 Neuron 7 Neuron 8 Neuron 36 Client 2
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 PFNM discovers correspondences among weights Client 1 Client 2 Client 1 Client 2 Client 2 Client 1 Neuron 12 Neuron 8 Neuron 49 Neuron 7 Neuron 36 Neuron 21 Matched neuron 8 Matched neuron 33 Matched neuron 44 Matched neuron 58
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 Summary PFNM is a method for combining pre-trained fully-connected neural networks: • Can combine NNs trained on heterogeneous data without access to data • Can be further improved with few communication rounds (if data is available) • Outperforms Distributed SGD and Federated Averaging Technical contributions: • Indian Buffet Process based model to govern correspondences between weights of local neural networks. Applicable to multilayer networks • BNP allows for adaptive learning of global NN size • Fast MAP inference using iterative Hungarian algorithm
IBM Research, MIT-IBM Watson AI Lab PFNM , Poster #20 THANK YOU | Please come to poster #2 #20 9

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Outline Motivation and challenge Dirichlet Process and Infinite Mixture Formulation

Nonparametric Bayesian M Nonparametric Bayesian Models odels --Learning and Reasoning in Open Possible Worlds -- Learning and Reasoning in Open Possible Worlds Eric Xing epxing@cs.cmu.edu Machine Learning Dept./Language Technology

783 views • 40 slides

Synergies in learning syllables and words or Adaptor grammars: a class of nonparametric Bayesian

Synergies in learning syllables and words or Adaptor grammars: a class of nonparametric Bayesian models Mark Johnson Brown University Joint work with Sharon Goldwater and Tom Griffiths NECPHON, November, 2008 1 / 26 Research goals Most

470 views • 26 slides

Dirichlet Processes and Nonparametric Bayesian Modelling Volker Tresp 1 Motivation Infinite

Dirichlet Processes and Nonparametric Bayesian Modelling Volker Tresp 1 Motivation Infinite models have recently gained a lot of attention in Bayesian machine learning They offer great flexibility and, in many applications, allow a more

1.84k views • 144 slides

Toward Reliable Bayesian Nonparametric Learning Erik Sudderth Brown University Department of

Toward Reliable Bayesian Nonparametric Learning Erik Sudderth Brown University Department of Computer Science Donglai Wei & Michael Bryant (HDP topics) Joint work with Michael Hughes & Emily Fox (BP-HMM) Documents & Topic Models

649 views • 61 slides

Nonparametric Bayesian Models for Sparse Matrices and Covariances Zoubin Ghahramani Department

Nonparametric Bayesian Models for Sparse Matrices and Covariances Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ Bayes 250 Edinburgh 2011 Bayesian Machine

402 views • 36 slides

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q Federated learning: FedSGD and FedAvg q Related research in federated learning q Open problems Outline q Preliminary: deep learning and SGD q Federated

616 views • 50 slides

Bayesian Nonparametric Models for Data Exploration Melanie F. Pradier Friday 15 th September,

Bayesian Nonparametric Models for Data Exploration Melanie F. Pradier Friday 15 th September, 2017 Intro BNPs ADDP for Marathon Modeling C-IBP for Clinical Trial PFAs for International Trade Conclusions Outline 1 Introduction 2 Bayesian

1.82k views • 118 slides

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression Bayesian Gaussian Mixture Models Non-parametric Bayes 2 Take Away ... 1. Maximum Likelihood Estimate (MLE) = arg max p ( D| )

609 views • 27 slides

Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo

Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo Chakraborty 2 , Prateek Mittal 1 and Seraphin Calo 2 1 Princeton University 2 IBM Research ICML 2019 Federated learning (with a malicious agent) Federated

1.46k views • 59 slides

Bayesian Meta-Learning CS 330 1 Logistics Homework 2 due next Wednesday. Project proposal due in

Bayesian Meta-Learning CS 330 1 Logistics Homework 2 due next Wednesday. Project proposal due in two weeks . Poster presentation: Tues 12/3 at 1:30 pm . 2 Disclaimers Bayesian meta-learning is an ac#ve area of research (like most of the

622 views • 33 slides

Bayesian nonparametric inference for diffusion models with discrete sampling Delft University of

Bayesian nonparametric inference for diffusion models with discrete sampling Delft University of Technology Jakob S ohl joint work with Richard Nickl Van Dantzig Seminar, Leiden, 26 October 2016 Jakob S ohl (TU Delft) Bayesian

514 views • 29 slides

Dr. Nonparametric Bayes Or: How I Learned to Stop Worrying and Love the Dirichlet Process Kurt

Dr. Nonparametric Bayes Or: How I Learned to Stop Worrying and Love the Dirichlet Process Kurt Miller CS 294: Practical Machine Learning November 19, 2009 Today we will discuss Nonparametric Bayesian methods. Kurt T. Miller Dr.

1.63k views • 124 slides

Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1

Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1 Outline Motivations Big data, IoT,AI Three vignettes: Federated machine learning Federated model aggregation Over-the-air

1.02k views • 56 slides

A Fast New Bayesian Approach to High-Dimensional Nonparametric Regression Without MCMC Ray Bai

A Fast New Bayesian Approach to High-Dimensional Nonparametric Regression Without MCMC Ray Bai Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Joint work with Gemma E. Moran (co-first author), Joseph

775 views • 55 slides

Fair Resource Allocation in Federated Learning Tian Li (CMU) , Maziar Sanjabi (Facebook AI), Ahmad

Fair Resource Allocation in Federated Learning Tian Li (CMU) , Maziar Sanjabi (Facebook AI), Ahmad Beirami (Facebook AI), Virginia Smith (CMU) tianli@cmu.edu 1 Federated Learning 2 2 Federated Learning Privacy-preserving training in

1.19k views • 55 slides

On a Class of Nonparametric Bayesian Autoregressive Models Maria Anna Di Lucca 1 , Alessandra

On a Class of Nonparametric Bayesian Autoregressive Models Maria Anna Di Lucca 1 , Alessandra Guglielmi 2 , uller 3 , Fernando A. Quintana 4 Peter M Karolinska Institutet 1 Politecnico di Milano 2 University of Texas, Austin 3 olica de Chile 4

1.02k views • 89 slides

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation

Bayesian Neural Networks - Presenters Group 1: A Practical Bayesian Framework for Backpropagation Networks - Slides 2-40 Paul Vicol Shane Baccas George Alexandru Adam Group 2: Priors for Infinite Networks - Slides 41-64

1.37k views • 110 slides

PDE models of neural networks Beno t Perthame Introduction The electrically active cells

PDE models of neural networks Beno t Perthame Introduction The electrically active cells are characterized by an action potential Hodgkin-Huxley FitzHugh-Nagumo Morris-Lekar Izhikevich Mitchell-Schaeffer Introduction 1

703 views • 29 slides

Lecture on Distributed Representations and Coarse Coding Geoffrey Hinton Localist

CSC321 Lecture on Distributed Representations and Coarse Coding Geoffrey Hinton Localist representations The simplest way to represent things with neural networks is to dedicate one neuron to each thing. Easy to understand. Easy to

374 views • 15 slides

DATA ANALYTICS USING DEEP LEARNING GT 8803 // FALL 2019 // JOY ARULRAJ L E C T U R E # 1 2 : T

DATA ANALYTICS USING DEEP LEARNING GT 8803 // FALL 2019 // JOY ARULRAJ L E C T U R E # 1 2 : T R A I N I N G N E U R A L N E T W O R K S ( P T 1 ) administrivia Reminders Integration with Eva Code reviews Each team must send

934 views • 70 slides

Deep Convolutional Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine

Deep Convolutional Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Deep Convolutional Neural Nets 1 / 25 Outline 1 Why Neural Networks? 2 Circuits 3 Neurons, Layers, and Networks 4 Correlation and Convolution 5

287 views • 25 slides

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORK ONLINE

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORK ONLINE COURSE http://info.usherbrooke.ca/hlarochelle/neural_networks Topics: online videos for a more detailed description of neural networks

1.4k views • 85 slides

Neural encoding models & maximum likelihood Jonathan Pillow 1 probability leftovers:

Statistical modeling and analysis of neural data NEU 560, Spring 2018 Lecture 7 Neural encoding models & maximum likelihood Jonathan Pillow 1 probability leftovers: sampling vs inference Model Data 700 samples sampling (measurement)

581 views • 30 slides

ResNet with one-neuron hidden layers is universal approximator Hongzhou Lin, Stefanie Jegelka

ResNet with one-neuron hidden layers is universal approximator Hongzhou Lin, Stefanie Jegelka Poster #28 In the 90s: Universal approximation theorem Output Hidden Layer Input . . . 1 hidden layer, width go to infinity universal

472 views • 9 slides