Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many - - PowerPoint PPT Presentation

▶
fairness ethics and machine learning
SMART_READER_LITE
LIVE PREVIEW

Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many - - PowerPoint PPT Presentation

Apr 12, 2023 184 likes •537 views

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/ Fairness, Ethics, and Machine Learning Prof. Mike Hughes Many ideas/slides attributable to: Alexandra Chouldechova Moritz Hardt 3 Fairness: Unit

slide-1
SLIDE 1

Fairness, Ethics, and Machine Learning

3

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2019s/

Many ideas/slides attributable to: Alexandra Chouldechova Moritz Hardt

  • Prof. Mike Hughes
slide-2
SLIDE 2

Fairness: Unit Objectives

  • How to think systematically about end-to-end ML
  • Where does data come from?
  • What features am I measuring? What protected

information can leak in unintentionally?

  • Who will be impacted?
  • How to define and measure notions fairness
  • Use concepts: accuracy, TPR, FPR, PPV, NPV
  • What is achievable? What is impossible?

4

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-3
SLIDE 3

Example Concerns about Fairness

5

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-4
SLIDE 4

6

Mike Hughes - Tufts COMP 135 - Spring 2019

Unfair image search

slide-5
SLIDE 5

Unfair Word Embeddings

7

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-6
SLIDE 6

Unfair Hiring?

slide-7
SLIDE 7

Job Ad Classifier: Is this fair?

slide-8
SLIDE 8

Unfair Recidivism Prediction

10

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-9
SLIDE 9

Focus: Binary Classifier

  • Let’s say we have two groups, A and B
  • Could be any protected group (race / gender / age)
  • We’re trying to build a binary classifier that will

predict individuals as HIGH or LOW risk

  • Likelihood of recidivism
  • Ability to pay back a loan

11

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-10
SLIDE 10

Group Discussion

  • When should protected information (gender,

race, age, etc) be provided as input to a predictor?

  • Can you build a “race-blind” classifier?
  • How could we measure if the predictions are

fair?

  • Is it enough to ensure accuracy parity?
  • ACC( group A) = ACC( group B )

12

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-11
SLIDE 11

Notation for Binary Classifier

13

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-12
SLIDE 12

Example of Accuracy Parity

14

Mike Hughes - Tufts COMP 135 - Spring 2019

Group A Group B true outcomes 1 = would fail to appear in court Y 0 0 1 1 0 0 1 1 classifier prediction 1 = too risky for bail C 0 0 0 0 1 1 1 1

Is this fair?

slide-13
SLIDE 13

Case Study: The COMPAS future crime prediction algorithm

15

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-14
SLIDE 14

16

Mike Hughes - Tufts COMP 135 - Spring 2019

HIGH RISK of future crime hold in jail before trial LOW RISK of future crime release before trial

COMPAS classifier

  • ther features (e.g. demographics, questionnaire

answers, family history)

slide-15
SLIDE 15

Mike Hughes - Tufts COMP 135 - Spring 2019

2

slide-16
SLIDE 16

18

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-17
SLIDE 17

The COMPAS tool assigns defendants scores from 1 to 10 that indicate how likely they are to reoffend based on more than 100 factors, including age, sex and criminal

  • history. Notably, race is not used. These scores

profoundly affect defendants’ lives: defendants who are defined as medium or high risk, with scores of 5-10, are more likely to be detained while awaiting trial than are low-risk defendants, with scores of 1-4.

19

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-18
SLIDE 18

20

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-19
SLIDE 19

21

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-20
SLIDE 20

22

Mike Hughes - Tufts COMP 135 - Spring 2019 Full Document: https://www.documentcloud.org/documents/2702103-Sample-Risk- Assessment-COMPAS-CORE.html

slide-21
SLIDE 21

23

Mike Hughes - Tufts COMP 135 - Spring 2019

ProPublica says: “Groups have different False Pos. Rates”

slide-22
SLIDE 22

Compas Team Says: “Groups have same predictive value”

24

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-23
SLIDE 23

False Positive Rate =

  • When true outcome is 0, how often does

classifier say “1”.

slide-24
SLIDE 24

True Positive Rate =

  • When true outcome is 1, how often does

classifier say “1”.

slide-25
SLIDE 25

Positive Predictive Value =

When classifier says “1”, how often is true label 1.

slide-26
SLIDE 26

Negative Predictive Value =

When classifier says “0”, how often is true label 0.

slide-27
SLIDE 27

29

Mike Hughes - Tufts COMP 135 - Spring 2019

ProPublica says: “Groups have different False Pos. Rates”

slide-28
SLIDE 28

Compas Team Says: “Groups have same predictive value”

30

Mike Hughes - Tufts COMP 135 - Spring 2019

slide-29
SLIDE 29

31

Mike Hughes - Tufts COMP 135 - Spring 2019

Worksheet

slide-30
SLIDE 30

32

Mike Hughes - Tufts COMP 135 - Spring 2019

Equation of the Day

FPR = p 1 − p 1 − PPV PPV TPR

where prevalence p = Pr(Y = 1)

If two groups have different p values, can we simultaneously have TPR parity AND FPR parity AND PPV parity AND NPV parity?

slide-31
SLIDE 31

33

Mike Hughes - Tufts COMP 135 - Spring 2019

https://www.propublica.org/article/bias-in-criminal-risk-scores- is-mathematically-inevitable-researchers-say

slide-32
SLIDE 32

34

Mike Hughes - Tufts COMP 135 - Spring 2019

Unless classifier is perfect, must chose one: Disparate Treatment (PPV or NPV not equal)

  • r Disparate Impact (FPR or TPR not equal)
slide-33
SLIDE 33

35

Mike Hughes - Tufts COMP 135 - Spring 2019

Try demo of making decisions from risk scores:

goo.gl/P8rmA3

slide-34
SLIDE 34

Fairness: Unit Objectives

  • How to think systematically about end-to-end ML
  • Where does data come from?
  • What features am I measuring? What protected

information can leak in unintentionally?

  • Who will be impacted?
  • How to define and measure notions fairness
  • Use concepts: accuracy, TPR, FPR, PPV, NPV
  • What is achievable? What is impossible?

36

Mike Hughes - Tufts COMP 135 - Spring 2019