fairness ethics and machine learning
play

Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many - PowerPoint PPT Presentation

Apr 12, 2023 •184 likes •536 views

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/ Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many ideas/slides attributable to: Alexandra Chouldechova Moritz Hardt 3 Fairness: Unit

  1. Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/ Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many ideas/slides attributable to: Alexandra Chouldechova Moritz Hardt 3

  2. Fairness: Unit Objectives • How to think systematically about end-to-end ML • Where does data come from? • What features am I measuring? What protected information can leak in unintentionally? • Who will be impacted? • How to define and measure notions fairness • Use concepts: accuracy, TPR, FPR, PPV, NPV • What is achievable? What is impossible? Mike Hughes - Tufts COMP 135 - Spring 2019 4

  3. Example Concerns about Fairness Mike Hughes - Tufts COMP 135 - Spring 2019 5

  4. Unfair image search Mike Hughes - Tufts COMP 135 - Spring 2019 6

  5. Unfair Word Embeddings Mike Hughes - Tufts COMP 135 - Spring 2019 7

  6. Unfair Hiring?

  7. Job Ad Classifier: Is this fair?

  8. Unfair Recidivism Prediction Mike Hughes - Tufts COMP 135 - Spring 2019 10

  9. Focus: Binary Classifier • Let’s say we have two groups, A and B • Could be any protected group (race / gender / age) • We’re trying to build a binary classifier that will predict individuals as HIGH or LOW risk • Likelihood of recidivism • Ability to pay back a loan Mike Hughes - Tufts COMP 135 - Spring 2019 11

  10. Group Discussion • When should protected information (gender, race, age, etc) be provided as input to a predictor? • Can you build a “race-blind” classifier? • How could we measure if the predictions are fair? • Is it enough to ensure accuracy parity? • ACC( group A) = ACC( group B ) Mike Hughes - Tufts COMP 135 - Spring 2019 12

  11. Notation for Binary Classifier Mike Hughes - Tufts COMP 135 - Spring 2019 13

  12. Example of Accuracy Parity Group A Group B true outcomes Y 0 0 1 1 0 0 1 1 1 = would fail to appear in court classifier prediction C 0 0 0 0 1 1 1 1 1 = too risky for bail Is this fair? Mike Hughes - Tufts COMP 135 - Spring 2019 14

  13. Case Study: The COMPAS future crime prediction algorithm Mike Hughes - Tufts COMP 135 - Spring 2019 15

  14. COMPAS classifier HIGH RISK of future crime hold in jail before trial LOW RISK of future crime release before trial other features (e.g. demographics, questionnaire answers, family history) Mike Hughes - Tufts COMP 135 - Spring 2019 16

  15. 2 Mike Hughes - Tufts COMP 135 - Spring 2019

  16. Mike Hughes - Tufts COMP 135 - Spring 2019 18

  17. The COMPAS tool assigns defendants scores from 1 to 10 that indicate how likely they are to reoffend based on more than 100 factors, including age, sex and criminal history. Notably, race is not used. These scores profoundly affect defendants’ lives: defendants who are defined as medium or high risk, with scores of 5-10, are more likely to be detained while awaiting trial than are low-risk defendants, with scores of 1-4. Mike Hughes - Tufts COMP 135 - Spring 2019 19

  18. Mike Hughes - Tufts COMP 135 - Spring 2019 20

  19. Mike Hughes - Tufts COMP 135 - Spring 2019 21

  20. Full Document: https://www.documentcloud.org/documents/2702103-Sample-Risk- Mike Hughes - Tufts COMP 135 - Spring 2019 Assessment-COMPAS-CORE.html 22

  21. ProPublica says: “Groups have different False Pos. Rates ” Mike Hughes - Tufts COMP 135 - Spring 2019 23

  22. Compas Team Says: “Groups have same predictive value ” Mike Hughes - Tufts COMP 135 - Spring 2019 24

  23. False Positive Rate = • When true outcome is 0, how often does classifier say “1”.

  24. True Positive Rate = • When true outcome is 1, how often does classifier say “1”.

  25. Positive Predictive Value = When classifier says “1”, how often is true label 1.

  26. Negative Predictive Value = When classifier says “0”, how often is true label 0.

  27. ProPublica says: “Groups have different False Pos. Rates ” Mike Hughes - Tufts COMP 135 - Spring 2019 29

  28. Compas Team Says: “Groups have same predictive value ” Mike Hughes - Tufts COMP 135 - Spring 2019 30

  29. Worksheet Mike Hughes - Tufts COMP 135 - Spring 2019 31

  30. Equation of the Day 1 − PPV p FPR = TPR 1 − p PPV where prevalence p = Pr(Y = 1) If two groups have different p values, can we simultaneously have TPR parity AND FPR parity AND PPV parity AND NPV parity? Mike Hughes - Tufts COMP 135 - Spring 2019 32

  31. https://www.propublica.org/article/bias-in-criminal-risk-scores- is-mathematically-inevitable-researchers-say Mike Hughes - Tufts COMP 135 - Spring 2019 33

  32. Unless classifier is perfect, must chose one: Disparate Treatment (PPV or NPV not equal) or Disparate Impact (FPR or TPR not equal) Mike Hughes - Tufts COMP 135 - Spring 2019 34

  33. Try demo of making decisions from risk scores: goo.gl/P8rmA3 Mike Hughes - Tufts COMP 135 - Spring 2019 35

  34. Fairness: Unit Objectives • How to think systematically about end-to-end ML • Where does data come from? • What features am I measuring? What protected information can leak in unintentionally? • Who will be impacted? • How to define and measure notions fairness • Use concepts: accuracy, TPR, FPR, PPV, NPV • What is achievable? What is impossible? Mike Hughes - Tufts COMP 135 - Spring 2019 36

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

INTRO TO ETHICS AND INTRO TO ETHICS AND FAIRNESS FAIRNESS Eunsuk Kang Required reading: R.

INTRO TO ETHICS AND INTRO TO ETHICS AND FAIRNESS FAIRNESS Eunsuk Kang Required reading: R.

INTRO TO ETHICS AND INTRO TO ETHICS AND FAIRNESS FAIRNESS Eunsuk Kang Required reading: R. Caplan, J. Donovan, L. Hanson, J. Matthews. "Algorithmic Accountability: A Primer", Data & Society (2018). 1 LEARNING GOALS LEARNING

732 views • 48 slides
ETHICS & FAIRNESS IN AI- ETHICS & FAIRNESS IN AI- ENABLED SYSTEMS ENABLED SYSTEMS

ETHICS & FAIRNESS IN AI- ETHICS & FAIRNESS IN AI- ENABLED SYSTEMS ENABLED SYSTEMS

ETHICS & FAIRNESS IN AI- ETHICS & FAIRNESS IN AI- ENABLED SYSTEMS ENABLED SYSTEMS Christian Kaestner (with slides from Eunsuk Kang) Required reading: R. Caplan, J. Donovan, L. Hanson, J. Matthews. " Algorithmic Accountability:

1.41k views • 75 slides
Fairness in Machine Learning Fairness in Supervised Learning Make decisions by machine learning:

Fairness in Machine Learning Fairness in Supervised Learning Make decisions by machine learning:

Fairness in Machine Learning Fairness in Supervised Learning Make decisions by machine learning: Software can make decision free of human biases Fairness in Supervised Learning Make decisions by machine learning: Software is

501 views • 46 slides
Fairness in Machine Learning: Part I Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Part I Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Part I Privacy & Fairness in Data Science CS848 Fall 2019 Outline High Level View Recap Supervised Learning: Binary Classification Survey of Approaches to Fairness in Supervised Learning Warmup:

593 views • 35 slides
Machine Learning for NLP Ethics and Machine Learning Aurlie Herbelot 2019 Centre for

Machine Learning for NLP Ethics and Machine Learning Aurlie Herbelot 2019 Centre for

Machine Learning for NLP Ethics and Machine Learning Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Today 1. Predicting or not predicting? That is the question. 2. Data and people: personalisation, bubbling,

1.07k views • 66 slides
Bias and Fairness in Machine Learning Irene Y. Chen @irenetrampoline

Bias and Fairness in Machine Learning Irene Y. Chen @irenetrampoline

Bias and Fairness in Machine Learning Irene Y. Chen @irenetrampoline http://gendershades.org/overview.html https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing COMPAS Correctional Offender Management

918 views • 48 slides
Fairness in Machine Learning: Practicum Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Practicum Privacy & Fairness in Data Science CS848 Fall 2019

Fairness in Machine Learning: Practicum Privacy & Fairness in Data Science CS848 Fall 2019 Human Decision Making Suppose we want to recommend a movie. Data Decision Maker Decision Jane likes Bollywood musicals. Bob: You should

3.14k views • 26 slides
Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice Schumann, Duncan C. McElfresh, John P. Dickerson, Michelle L. Mazurek, Michael Carl Tschantz 37th International Conference on Machine Learning (ICML)

980 views • 69 slides
Bias, Fairness, Accountability, and Transparency in Machine Learning CS 115 Computing for the

Bias, Fairness, Accountability, and Transparency in Machine Learning CS 115 Computing for the

Bias, Fairness, Accountability, and Transparency in Machine Learning CS 115 Computing for the Socio-Techno Web Instructor: Brian Brubach Announcements Adjustment to deadline schedule Assignment 5 due Tuesday Project milestone 4 due

845 views • 22 slides
15: Ethics in Machine Learning, plus Artificial General Intelligence and some old Science Fiction

15: Ethics in Machine Learning, plus Artificial General Intelligence and some old Science Fiction

15: Ethics in Machine Learning, plus Artificial General Intelligence and some old Science Fiction Machine Learning and Real-world Data Ann Copestake and Simone Teufel Computer Laboratory University of Cambridge Lent 2017 Some ethical issues

403 views • 21 slides
Fairness and bias in Machine Learning A quick review on tools to detect biases in machine

Fairness and bias in Machine Learning A quick review on tools to detect biases in machine

QCon 2019 Fairness and bias in Machine Learning A quick review on tools to detect biases in machine learning model Thierry Silbermann, Tech Lead Data Science at Nubank thierry.silbermann@nubank.com.br Data collection Todays

870 views • 45 slides
Fairness and Discrimination in Mechanism Design and Machine Learning Jessie Finocchiaro 1 ,

Fairness and Discrimination in Mechanism Design and Machine Learning Jessie Finocchiaro 1 ,

Fairness and Discrimination in Mechanism Design and Machine Learning Jessie Finocchiaro 1 , Roland Maio 2 , Faidra Monachou 3 Gourab K Patro 4 , Manish Raghavan 5 , Ana-Andreea Stoica 2 , Stratis Tsirtis 6 1 Colorado 2 Columbia 3 Stanford 4 IIT

521 views • 14 slides
Recent Results on Algorithmic Fairness and Meta-Learning Massimiliano Pontil Computational

Recent Results on Algorithmic Fairness and Meta-Learning Massimiliano Pontil Computational

Recent Results on Algorithmic Fairness and Meta-Learning Massimiliano Pontil Computational Statistics and Machine Learning Istituto Italiano di Tecnologia and Department of Computer Science University College London 4th Annual Machine

439 views • 26 slides
Machine Learning 2 DS 4420 - Spring 2020 Bias and fairness Byron C. Wallace Material in this

Machine Learning 2 DS 4420 - Spring 2020 Bias and fairness Byron C. Wallace Material in this

Machine Learning 2 DS 4420 - Spring 2020 Bias and fairness Byron C. Wallace Material in this lecture modified from materials created by Jay Alammar (http://jalammar.github.io/ illustrated-transformer/) and Sasha Rush

1.6k views • 79 slides
FairSquare: Probabilistic Verification of Program Fairness Aws Albarghouthi Loris DAntoni

FairSquare: Probabilistic Verification of Program Fairness Aws Albarghouthi Loris DAntoni

FairSquare: Probabilistic Verification of Program Fairness Aws Albarghouthi Loris DAntoni Samuel Drews University of Wisconsin-Madison Aditya V. Nori Microsoft Research Machine Data Model Learning Machine Data Model Learning small

654 views • 39 slides
arXiv:1710.06921v1 [cs.CY] 18 Oct 2017 ABSTRACT and fairness-aware ML methods [6, 7, 8, 9, 10,

arXiv:1710.06921v1 [cs.CY] 18 Oct 2017 ABSTRACT and fairness-aware ML methods [6, 7, 8, 9, 10,

Themis-ml: A Fairness-aware Machine Learning Interface for End-to-end Discrimination Discovery and Mitigation Niels Bantilan Arena.io New York, NY niels.bantilan@gmail.com arXiv:1710.06921v1 [cs.CY] 18 Oct 2017 ABSTRACT and fairness-aware

311 views • 8 slides
1 Aspects of Search Quality System Aspects of Evaluation Relevancy Response time:

1 Aspects of Search Quality System Aspects of Evaluation Relevancy Response time:

Table of Content Search Engine Evaluation Metrics for relevancy Precision/recall Search Evaluation F-measure MAP NDCG Tao Yang CS290N Slides partially based on text book [CMS] [MRS] Difficulties in Evaluating IR Systems

492 views • 8 slides
Finance Review Ivo Welch January 6, 2019 Introduction Write your section number, row

Finance Review Ivo Welch January 6, 2019 Introduction Write your section number, row

Finance Review Ivo Welch January 6, 2019 Introduction Write your section number, row number, and initials. If you do not know the answer to a question, write do not know and go on. If the question is quickly answerable,

345 views • 9 slides
MA111: Contemporary mathematics . Jack Schmidt University of Kentucky August 27, 2012 Entrance

MA111: Contemporary mathematics . Jack Schmidt University of Kentucky August 27, 2012 Entrance

. MA111: Contemporary mathematics . Jack Schmidt University of Kentucky August 27, 2012 Entrance Slip (due 5 min past the hour): Convert the following linear ballots into a preference schedule: Ballot Ballot Ballot Ballot Ballot Ballot

356 views • 17 slides
3 COMP 1 5 9 3 Algorithmic Verification Safety and Liveness, Fairness Dr. Liam OConnor

3 COMP 1 5 9 3 Algorithmic Verification Safety and Liveness, Fairness Dr. Liam OConnor

<latexit

851 views • 66 slides
COMP31212: Concurrency Topic 5.3: Liveness and Topic 5.4 Fairness Topic 5.3: Liveness Properties

COMP31212: Concurrency Topic 5.3: Liveness and Topic 5.4 Fairness Topic 5.3: Liveness Properties

Topic 5.3: Liveness Properties Topic 5.4: Fairness and Starvation COMP31212: Concurrency Topic 5.3: Liveness and Topic 5.4 Fairness Topic 5.3: Liveness Properties Topic 5.4: Fairness and Starvation Outline Topic 5.3: Liveness Properties

509 views • 34 slides
Fairness in reactive programming Andrew Cave McGill University January 25, 2013 Joint work with

Fairness in reactive programming Andrew Cave McGill University January 25, 2013 Joint work with

Fairness in reactive programming Andrew Cave McGill University January 25, 2013 Joint work with Francisco Ferreira, Prakash Panangaden and Brigitte Pientka 1 / 14 Introduction Reactive programming languages provide a setting to implement

307 views • 14 slides
On the cost of essentially fair clusterings Ioana Bercea, Martin Gro, Samir Khuller, Aounon

On the cost of essentially fair clusterings Ioana Bercea, Martin Gro, Samir Khuller, Aounon

On the cost of essentially fair clusterings Ioana Bercea, Martin Gro, Samir Khuller, Aounon Kumar, Clemens Rsner, Daniel Schmidt, Melanie Schmidt Aussois, January 2019 1 Assigning students to schools Example problem Goals Minimize

981 views • 25 slides
P4air: Increasing Fairness among Competing Congestion Control Algorithms Belma Turkovic and

P4air: Increasing Fairness among Competing Congestion Control Algorithms Belma Turkovic and

P4air: Increasing Fairness among Competing Congestion Control Algorithms Belma Turkovic and Fernando Kuipers ICNP 2020, October 13-16, 2020 Congestion control algorithms Congestion control algorithms - New protocols and congestion control

1.13k views • 102 slides

More recommend