fair resource allocation in federated learning
play

Fair Resource Allocation in Federated Learning Tian Li (CMU) , - PowerPoint PPT Presentation

Jan 27, 2023 •619 likes •1.19k views

Fair Resource Allocation in Federated Learning Tian Li (CMU) , Maziar Sanjabi (Facebook AI), Ahmad Beirami (Facebook AI), Virginia Smith (CMU) tianli@cmu.edu 1 Federated Learning 2 2 Federated Learning Privacy-preserving training in

  1. Fair Resource Allocation in Federated Learning Tian Li (CMU) , Maziar Sanjabi (Facebook AI), Ahmad Beirami (Facebook AI), Virginia Smith (CMU) tianli@cmu.edu 1

  2. Federated Learning 2 2

  3. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2 2

  4. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2 2

  5. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2 2

  6. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  7. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  8. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  9. Federated Learning Privacy-preserving training in heterogeneous, (potentially) massive networks 2

  10. Challenges 3

  11. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w 3

  12. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w no accuracy guarantee for individual devices 3

  13. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w no accuracy guarantee for individual devices model performance can vary widely 3

  14. Challenges ) … ( objective: + p 2 min p 1 + p N + F 1 F 2 F N w no accuracy guarantee for individual devices Can we devise an efficient federated optimization method to encourage a more fair (i.e., more uniform ) distribution of the model performance across devices? model performance can vary widely 3

  15. Fair Resource Allocation Objective

  16. Fair Resource Allocation Objective ) … ( + p 2 min p 1 + p N + F 1 F 2 F N w

  17. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w

  18. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks

  19. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞

  20. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞ Theory

  21. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞ Theory

  22. Fair Resource Allocation Objective ) q + 1 … q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N q + 1 w Inspired by 𝛽 -fairness for fair resource allocation in wireless networks A tunable framework ( : previous objective; : minimax fairness) q = 0 q = ∞ Theory Generalization guarantees Increasing results in more ‘uniform’ accuracy distributions (in q terms of various uniformity measures such as variance)

  23. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w

  24. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w # test accuracy 0.4 0.2 0.6 0.8

  25. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w Baseline # test accuracy 0.4 0.2 0.6 0.8

  26. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w Baseline q -FFL # test accuracy 0.4 0.2 0.6 0.8

  27. Fair Resource Allocation Objective ) … q + 1 q + 1 q + 1 ( q + 1 1 q -FFL: + p 2 min p 1 + p N + F 1 F 2 F N w Baseline q -FFL # test accuracy 0.4 0.2 0.6 0.8

  28. Efficient Solver

  29. Efficient Solver Challenges

  30. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s

  31. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s

  32. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  33. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  34. Efficient Solver Challenges Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  35. Efficient Solver Challenges High level ideas Different fairness/accuracy tradeoffs: different q’s Heterogeneous networks, expensive communication

  36. Efficient Solver Challenges High level ideas Different fairness/accuracy Dynamically estimate the step tradeoffs: different q’s sizes associated with different ’s q Heterogeneous networks, expensive communication

  37. Efficient Solver Challenges High level ideas Different fairness/accuracy Dynamically estimate the step tradeoffs: different q’s sizes associated with different ’s q Heterogeneous networks, expensive communication

  38. Efficient Solver Challenges High level ideas Different fairness/accuracy Dynamically estimate the step tradeoffs: different q’s sizes associated with different ’s q Heterogeneous networks, Allow for low device expensive communication participation, local updating

  39. Empirical Results Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespeare q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  40. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespeare q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  41. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  42. Empirical Results Benchmark: similar average LEAF (leaf.cmu.edu) accuracy Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  43. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  44. Empirical Results Benchmark: decrease variance LEAF (leaf.cmu.edu) significantly Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  45. Empirical Results Benchmark: LEAF (leaf.cmu.edu) Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  46. Empirical Results increase the Benchmark: accuracy of the LEAF (leaf.cmu.edu) worst 10% devices Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

  47. Empirical Results slightly decrease Benchmark: the accuracy of the LEAF (leaf.cmu.edu) best devices Objective Average Worst 10% Best 10% Variance Dataset Synthetic q = 0 80.8 18.8 100.0 724 q = 1 79.0 31.1 100.0 472 Vehicle q = 0 87.3 43.0 95.7 291 q = 5 87.7 69.9 94.0 48 Sent140 q = 0 65.1 15.9 100.0 697 q = 1 66.5 23.0 100.0 509 Shakespe q = 0 51.1 39.7 72.9 82 q = .001 52.1 42.1 69.0 54

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Dominant Resource Fairness: Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness: Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness: Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, Ion Stoica UC Berkeley Introduction: Resource Allocation There is a need to design fair,

604 views • 22 slides
D Dominant Resource Fairness (DRF) i t R F i (DRF) Fair Allocation of Multiple Resource Types

D Dominant Resource Fairness (DRF) i t R F i (DRF) Fair Allocation of Multiple Resource Types

D Dominant Resource Fairness (DRF) i t R F i (DRF) Fair Allocation of Multiple Resource Types Ali Ghodsi , Matei Zaharia Benjamin Hindman, Andy Konwinski, B j i Hi d A d K i ki Scott Shenker, Ion Stoica University of California, Berkeley

781 views • 51 slides
Dominant Resource Fairness : Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness : Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei

Dominant Resource Fairness : Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, Ion Stoica University of California, Berkeley Talk by: Yael Amsterdamer Introduction Why

800 views • 22 slides
Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg Shah Microsoft Research

Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg Shah Microsoft Research

Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg Shah Microsoft Research New York University of Toronto EC'19, AAAI'20 and AAMAS'20 Tutorial on Recent Advances in Fair Resource Allocation Rupert Freeman and Nisarg

2.07k views • 160 slides
Max-Min Fair Resource Allocation for Multiuser Amplify-and-Forward Relay Networks Alireza

Max-Min Fair Resource Allocation for Multiuser Amplify-and-Forward Relay Networks Alireza

Max-Min Fair Resource Allocation for Multiuser Amplify-and-Forward Relay Networks Alireza Sharifian*, Petar Djukic*, Halim Yanikomeroglu*, and Jietao Zhang *Department of Computer and System engineering Carleton University Huawei Wireless

466 views • 21 slides
Fair and Sustainable Resource Allocation in Self-Organising Multi-Agent Systems Jeremy Pitt

Fair and Sustainable Resource Allocation in Self-Organising Multi-Agent Systems Jeremy Pitt

Fair and Sustainable Resource Allocation in Self-Organising Multi-Agent Systems Jeremy Pitt Department of Electrical and Electronic Engineering Journ ees Francophones sur les Syst` emes Multi-Agents (JFSMA) Plate-forme Intelligence

619 views • 22 slides
Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q Federated learning: FedSGD and FedAvg q Related research in federated learning q Open problems Outline q Preliminary: deep learning and SGD q Federated

616 views • 50 slides
FAT in Resource Allocation: Equilibrium to the Rescue Noam Nisan Hebrew U. Computer Science

FAT in Resource Allocation: Equilibrium to the Rescue Noam Nisan Hebrew U. Computer Science

FAT in Resource Allocation: Equilibrium to the Rescue Noam Nisan Hebrew U. Computer Science Moshe Babaioff Microsoft Research Inbal Talgam-Cohen Technion Computer Science (+ LL.B.) (Based on our paper Fair Allocation through

664 views • 30 slides
Section 6.1: Resource Allocation Issues Chapter 6: Congestion Control and Resource Allocation

Section 6.1: Resource Allocation Issues Chapter 6: Congestion Control and Resource Allocation

Section 6.1: Resource Allocation Issues Chapter 6: Congestion Control and Resource Allocation CS/ECPE 5516: Comm. Network Prof. Abrams Spring 2000 CS/ECPE5516 1 2 CS/ECPE5516 Whats wrong with doing How to prevent traffic jams

311 views • 6 slides
Machine Learning as Enabler for Cross-Layer Resource Allocation: Opportunities and Challenges

Machine Learning as Enabler for Cross-Layer Resource Allocation: Opportunities and Challenges

| 1 Machine Learning as Enabler for Cross-Layer Resource Allocation: Opportunities and Challenges with Deep Reinforcement Learning Fatemeh Shah-Mohammadi and Andres Kwasinski Rochester Institute of Technology | 2 Outline Benefits for

524 views • 18 slides
Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo

Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo

Analyzing Federated Learning through an Adversarial Lens Arjun Nitin Bhagoji 1 , Supriyo Chakraborty 2 , Prateek Mittal 1 and Seraphin Calo 2 1 Princeton University 2 IBM Research ICML 2019 Federated learning (with a malicious agent) Federated

1.46k views • 59 slides
On Self-adaptive Resource Allocation through Reinforcement Learning Jacopo Panerati , Filippo

On Self-adaptive Resource Allocation through Reinforcement Learning Jacopo Panerati , Filippo

On Self-adaptive Resource Allocation through Reinforcement Learning Jacopo Panerati , Filippo Sironi , Matteo Carminati , Martina Maggio , Giovanni Beltrame , Piotr J. Gmytrasiewicz , Donatella Sciuto and Marco D.

1.33k views • 109 slides
Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1

Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1

Federated Machine Learning via Over-the-Air Computation Yuanming Shi ShanghaiTech University 1 Outline Motivations Big data, IoT,AI Three vignettes: Federated machine learning Federated model aggregation Over-the-air

1.02k views • 56 slides
Resource Allocation Task Force Resource Allocation Task Force Gigi Karmous Edwards Gigi

Resource Allocation Task Force Resource Allocation Task Force Gigi Karmous Edwards Gigi

Resource Allocation Task Force Resource Allocation Task Force Gigi Karmous Edwards Gigi ke@ncsu.edu Gigi_ke@ncsu.edu GLIF Winter Meeting, Hong Kong 2011 Purpose of R Resource Allocation Task Force All i T k F What does the user want

342 views • 7 slides
Resource Allocation and Deadlock Resource Allocation and Deadlock Handling Conditions for

Resource Allocation and Deadlock Resource Allocation and Deadlock Handling Conditions for

Resource Allocation and Deadlock Resource Allocation and Deadlock Handling Conditions for Deadlock [Coffman-etal 1971] 4 conditions must hold simultaneously for a deadlock to occur: deadlock to occur: Mutual exclusion: only one process at

520 views • 31 slides
Fair Allocation of Indivisible Goods Ulle Endriss Institute for Logic, Language and Computation

Fair Allocation of Indivisible Goods Ulle Endriss Institute for Logic, Language and Computation

Fair Allocation of Indivisible Goods T ubingen, 7 April 2016 Fair Allocation of Indivisible Goods Ulle Endriss Institute for Logic, Language and Computation University of Amsterdam Ulle Endriss 1 Fair Allocation of Indivisible Goods T

568 views • 20 slides
FACEBOOK & CAMBRIDGE ANALYTICAS DATA BREACH By Vignesh Kumar and Liu Jason Tan OUTLINE

FACEBOOK & CAMBRIDGE ANALYTICAS DATA BREACH By Vignesh Kumar and Liu Jason Tan OUTLINE

FACEBOOK & CAMBRIDGE ANALYTICAS DATA BREACH By Vignesh Kumar and Liu Jason Tan OUTLINE Summary Initial aftermath Cambridge Analytica Zuckerbergs response Global Science Research Zuckerbergs testimony before

618 views • 26 slides
What well cover Why a FB Page? o Benefits o Limitations of other approaches How to stay

What well cover Why a FB Page? o Benefits o Limitations of other approaches How to stay

What well cover Why a FB Page? o Benefits o Limitations of other approaches How to stay out of jail Stories vs. Posts Scheduling Insights Quick Tour of the page Finding Your Target Customers Who are my target customers?

480 views • 13 slides
Landmark Labelling for 3D Faces Clement Creusot, Nick Pears, Jim Austin Clment Creusot, October

Landmark Labelling for 3D Faces Clement Creusot, Nick Pears, Jim Austin Clment Creusot, October

Landmark Labelling for 3D Faces Clement Creusot, Nick Pears, Jim Austin Clment Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 1 Table of Content Motivation Motivation Problem Problem Solution Solution Results

540 views • 39 slides
Computer Vision to Measure Life Measure Life Tilo Burghardt Personal Background Graduation

Computer Vision to Measure Life Measure Life Tilo Burghardt Personal Background Graduation

Computer Vision to Measure Life Measure Life Tilo Burghardt Personal Background Graduation in Media Computing at Dresden University of Technology Emigration to the United Kingdom Postgraduate Study and Fellowship at the University

82 views • 5 slides
Anti-Muslim Hate Thriving on Facebook NJ School Board official targets Reps. Rashida Tlaib &

Anti-Muslim Hate Thriving on Facebook NJ School Board official targets Reps. Rashida Tlaib &

Anti-Muslim Hate Thriving on Facebook NJ School Board official targets Reps. Rashida Tlaib & Ilhan Omar Tennessee County Commissioner Anti-Muslim proponent Jon Ritzheimer threatens Muslim community and calls others to action Russians

418 views • 3 slides
Adversarial Continual Learning Sayna Ebrahimi Franziska Meier Roberto Calandra Trevor Darrell

Adversarial Continual Learning Sayna Ebrahimi Franziska Meier Roberto Calandra Trevor Darrell

Adversarial Continual Learning Sayna Ebrahimi Franziska Meier Roberto Calandra Trevor Darrell Marcus Rohrbach UC Berkeley Facebook AI Research Facebook AI Research UC Berkeley Facebook AI Research What is Continual Learning? Definition:

592 views • 35 slides
Welcome to a Power On Network Online Presentation From CNN to Facebook: Harnessing the Power of

Welcome to a Power On Network Online Presentation From CNN to Facebook: Harnessing the Power of

Welcome to a Power On Network Online Presentation From CNN to Facebook: Harnessing the Power of Integrated Communications with Joseph Phelan at Miami Workers Center ReadyTalk Call info: 1-866-740-1260 Access code: 7242600

649 views • 31 slides
The Spread of Preemption And How to Fight It Tallahassee Mayor Andrew Gillum

The Spread of Preemption And How to Fight It Tallahassee Mayor Andrew Gillum

The Spread of Preemption And How to Fight It Tallahassee Mayor Andrew Gillum Twitter.com/DefendLocal Facebook.Com/DefendLocal DefendLocal.com Attack By the Gun Lobby Mayor Gillum and City

243 views • 9 slides

More recommend