FAIRNESS RISK MEASURES FAIRNESS RISK MEASURES Robert C. Williamson - PowerPoint PPT Presentation

FAIRNESS RISK MEASURES

FAIRNESS RISK MEASURES Robert C. Williamson Aditya Menon

LOSS FUNCTIONS - OUTCOME CONTINGENT UTILITIES

LOSS FUNCTIONS - OUTCOME CONTINGENT UTILITIES ▸ Wald’s abstraction: a loss function ℓ : Y × A → ℝ + ∪ {+ ∞ } =: ℝ Label 
 Action 
 space space

LOSS FUNCTIONS - OUTCOME CONTINGENT UTILITIES ▸ Wald’s abstraction: a loss function ℓ : Y × A → ℝ + ∪ {+ ∞ } =: ℝ Label 
 Action 
 space space ▸ is an outcome contingent utility a ↦ ℓ ( y , a )

LOSS FUNCTIONS - OUTCOME CONTINGENT UTILITIES ▸ Wald’s abstraction: a loss function ℓ : Y × A → ℝ + ∪ {+ ∞ } =: ℝ Label 
 Action 
 space space ▸ is an outcome contingent utility a ↦ ℓ ( y , a ) ▸ Learning goal: expected risk minimisation min f ∈ℱ 𝔽 ( 𝖸 , 𝖹 ) ∼ P ℓ ( 𝖹 , f ( 𝖸 ))

LOSS FUNCTIONS - OUTCOME CONTINGENT UTILITIES ▸ Wald’s abstraction: a loss function ℓ : Y × A → ℝ + ∪ {+ ∞ } =: ℝ Label 
 Action 
 space space ▸ is an outcome contingent utility a ↦ ℓ ( y , a ) ▸ Learning goal: expected risk minimisation min f ∈ℱ 𝔽 ( 𝖸 , 𝖹 ) ∼ P ℓ ( 𝖹 , f ( 𝖸 )) ▸ In practice: empirical risk minimisation min f ∈ℱ 𝔽 ( 𝖸 , 𝖹 ) ∼ P m ℓ ( 𝖹 , f ( 𝖸 )) m 1 ∑ = min ℓ ( y i , f ( x i )) m f ∈ℱ i =1

MINIMISING EMPIRICAL RISK

MINIMISING EMPIRICAL RISK 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS

MINIMISING EMPIRICAL RISK 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F1 F1

MINIMISING EMPIRICAL RISK 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F2 F2

MINIMISING EMPIRICAL RISK 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F3 F3

MINIMISING EMPIRICAL RISK 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F4 F4

MINIMISING EMPIRICAL RISK 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F5 F5

MINIMISING EMPIRICAL RISK 200 200 150 150 Loss 100 100 Average 
 loss of 
 50 50 best 
 hypothesis 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F5 F5

MINIMISING EMPIRICAL RISK WITH SENSITIVE ATTRIBUTES VISIBLE

MINIMISING EMPIRICAL RISK WITH SENSITIVE ATTRIBUTES VISIBLE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS

MINIMISING EMPIRICAL RISK WITH SENSITIVE ATTRIBUTES VISIBLE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F1 F1

MINIMISING EMPIRICAL RISK WITH SENSITIVE ATTRIBUTES VISIBLE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F2 F2

MINIMISING EMPIRICAL RISK WITH SENSITIVE ATTRIBUTES VISIBLE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F3 F3

MINIMISING EMPIRICAL RISK WITH SENSITIVE ATTRIBUTES VISIBLE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F4 F4

MINIMISING EMPIRICAL RISK WITH SENSITIVE ATTRIBUTES VISIBLE 200 200 150 150 Loss 100 100 Average 
 loss of 
 50 50 best 
 hypothesis 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 Data index HYPOTHESIS F5 F5

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 100 100 50 50 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F1 F1

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F2 F2

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F3 F3

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F4 F4

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 Average 
 loss of 
 50 50 best 
 hypothesis 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F5 F5

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 Average 
 loss of 
 50 50 best 
 hypothesis 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F5 F5

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 Average 
 loss of 
 50 50 best 
 hypothesis 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F5 F5

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 50 50 0 0 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 HYPOTHESIS F5 F5

MINIMISING EMPIRICAL RISK REINDEXING PER SENSITIVE ATTRIBUTE 200 200 150 150 Loss 100 100 50 50 0 0 1 2 3 1 1 4 4 7 7 10 10 2 2 6 6 8 8 3 3 5 5 9 9 Sensitive 
 Feature index HYPOTHESIS F5 F5

MINIMISING AGGREGATED EMPIRICAL RISK

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 150 150 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 150 150 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F1 F1 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 150 150 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F2 F2 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 150 150 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F3 F3 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 150 150 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F4 F4 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 150 150 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F5 F5 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 ▸ Standard problem: minimise average risk 150 150 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F5 F5 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 ▸ Standard problem: minimise average risk 150 150 ▸ Equity problem: also take account of variation 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F5 F5 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK 200 200 ▸ Standard problem: minimise average risk 150 150 ▸ Equity problem: also take account of variation Loss ▸ Fairness problem: mixture of both 100 100 50 50 0 0 Sensitive 
 1 1 2 2 3 3 Feature index F5 F5 HYPOTHESIS

MINIMISING AGGREGATED EMPIRICAL RISK AND DEVIATION

MINIMISING AGGREGATED EMPIRICAL RISK AND DEVIATION ▸ Trade off low deviation against higher average

MINIMISING AGGREGATED EMPIRICAL RISK AND DEVIATION 80 80 ▸ Trade off low deviation against higher average 60 60 40 40 Loss 20 20 0 0 1 1 2 2 3 3

MINIMISING AGGREGATED EMPIRICAL RISK AND DEVIATION 80 80 ▸ Trade off low deviation against higher average 60 60 40 40 Loss 20 20 0 0 1 1 2 2 3 3 F5 F5

MINIMISING AGGREGATED EMPIRICAL RISK AND DEVIATION 80 80 ▸ Trade off low deviation against higher average 60 60 40 40 Loss 20 20 0 0 1 1 2 2 3 3 F6 F6

MINIMISING AGGREGATED EMPIRICAL RISK AND DEVIATION 80 80 ▸ Trade off low deviation against higher average 60 60 ▸ Let be the sensitive feature space S = {1,2,3} 40 40 Loss 20 20 0 0 1 1 2 2 3 3 F6 F6

MINIMISING AGGREGATED EMPIRICAL RISK AND DEVIATION 80 80 ▸ Trade off low deviation against higher average 60 60 ▸ Let be the sensitive feature space S = {1,2,3} ▸ For let be a r.v. (taking as the 𝖲 f : S → ℝ f ∈ ℱ S 40 40 sample space, with a uniform base measure) Loss 20 20 𝖲 f : S ∋ s ↦ 𝔽 ( 𝖸 , 𝖹 ) [ ℓ ( 𝖹 , f ( 𝖸 )) | 𝖳 = s ] 0 0 1 1 2 2 3 3 F6 F6