Leveraging GANs for fairness evaluations Emily Denton Research - - PowerPoint PPT Presentation
Leveraging GANs for fairness evaluations Emily Denton Research Scientist, Google Brain Emily Denton Margaret Mitchell Timnit Gebru Ben Hutchinson Background ML Fairness seeks to address algorithmic unfairness , with a focus on machine
Ben Hutchinson Margaret Mitchell Timnit Gebru Emily Denton Emily Denton Research Scientist, Google Brain
P • 2
Very broad research area! I will be focusing on one specifjc component: detecting undesirable bias in computer vision systems
P • 3
The Coded Gaze: Unmasking Algorithmic Bias Joy Buolamwini
Unrepresentative training data can lead to disparities in accuracy for difgerent demographics
P • 4
[Wilson et al. Predictive inequity in object detection. arXiv:1902.11097, 2019]
P • 5
[Zhao et al. Men Also Like Shopping: Reducing Gender Bias Amplifjcation using Corpus-level Constraints. EMNLP, 2017.]
Social biases embedded in data distribution can be reproduced and/or amplifjed
[Hendricks et al. Women also snowboard: Overcoming bias in captioning models. ECCV, 2018]
P • 6
[Misra et al. Seeing through the Human Reporuing Bias: Visual Classifjers from Noisy Human-Centric Labels. CVPR 2016]
Human reporuing bias can afgect annotations
P • 7
[Misra et al. Seeing through the Human Reporuing Bias: Visual Classifjers from Noisy Human-Centric Labels. CVPR 2016]
Human reporuing bias can afgect annotations
“Green bananas”
P • 8
“doctor” “female doctor”
“nurse”
[Misra et al. Seeing through the Human Reporuing Bias: Visual Classifjers from Noisy Human-Centric Labels. CVPR 2016]
Social biases can afgect annotations and propagate through ML system
P • 9
Social biases can afgect annotations and propagate through ML system
[Rhue. Racial Infmuence on Automated Perceptions of Emotions. 2019]
P • 10
High quality photo realistic images
[Karras et al. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018]
P • 11
High quality photo realistic images
[Karras et al. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018]
Controllable image synthesis
P • 12
Generative techniques provide tools for testing a classifjer’s sensitivity to difgerent image features Can answer questions of the form: How does the classifjer’s output change as some characteristic of the image is systematically varied? Is the classifjer sensitive to a characteristic that should be irrelevant for the task?
P • 13
CNN
P(Smile | x)
x
P • 14
CNN
P(Smile | x)
x x’
Manipulate facial hair
P • 15
CNN
P(Smile | x)
x
CNN
P(Smile | x’)
x’
Did it change?
Manipulate facial hair
P • 16
Can observe the efgect on a classifjers of systematically manipulating factors of variation in an image
P • 17
All else being equal, the presence of facial hair should be irrelevant to the classifjer
Can observe the efgect on a classifjers of systematically manipulating factors of variation in an image
P • 18
Smiling classifjer trained on CelebA (128x128 resolution images)
P • 19
Smiling classifjer trained on CelebA (128x128 resolution images) Standard progressive GAN trained to generate 128x128 CelebA images
P • 20
Smiling classifjer trained on CelebA (128x128 resolution images) Standard progressive GAN trained to generate 128x128 CelebA images Encoder trained to infer latent codes that generated an images
P • 21
Directions in latent space that manipulate a paruicular factor of variation in the image
Latent codes corresponding to images without aturibute a Latent codes corresponding to images with aturibute a Aturibute vector
P • 22
Eyeglasses = 1 Eyeglasses = 0 Mustache = 1 Mustache = 0 Blond_Hair = 1 Blond_Hair = 0
We infer aturibute vectors using binary CelebA annotations
P • 23
Mustache = 1 Mustache = 0
P • 24
Many of the atuributes are subjective or ill-defjned Interpretation of category boundaries is contingent on the annotators The resulting manipulations refmect how the paruicular atuributes were operationalized and measured within the CelebA dataset
P • 25
P • 26
P • 27
P • 28
P • 29
Model f outputs the probability of a smile being present in the image: Sensitivity of the continuous valued output of f to changes defjned by the aturibute vector d:
Difgerence in classifjers’ output that results from moving in direction d in latent space
P • 30
Given a threshold, 0 ≤ c ≤ 1, binary classifjcations are
Sensitivity of the discrete classifjcation decision to peruurbations along an vector d as:
Frequency with which classifjcation fmips from smiling to not smiling Frequency with which classifjcation fmips from not smiling to smiling
P • 31
Given a threshold, 0 ≤ c ≤ 1, binary classifjcations are
Sensitivity of the discrete classifjcation decision to peruurbations along an vector d as:
P • 32
Given a threshold, 0 ≤ c ≤ 1, binary classifjcations are
Sensitivity of the discrete classifjcation decision to peruurbations along an vector d as:
P • 33
Given a threshold, 0 ≤ c ≤ 1, binary classifjcations are
Sensitivity of the discrete classifjcation decision to peruurbations along an vector d as:
P • 34
~12% of images initially classifjed as not smiling get classifjed as smiling afuer Heavy_Makeup augmentation
P • 35
~12% of images initially classifjed as not smiling get classifjed as smiling afuer Heavy_Makeup augmentation
P • 36
~7% of images initially classifjed as smiling get classifjed as not smiling afuer Young augmentation
P • 37
BUT, need to be careful the aturibute vector hasn’t actually encoded something that should be relevant to smiling classifjcation!
Mouth expression has defjnitely changed ~40% of images initially classifjed as not smiling get classifjed as smiling afuer High_Cheekbones augmentation
P • 38
BUT, need to be careful the aturibute vector hasn’t actually encoded something that should be relevant to smiling classifjcation!
So far we’re verifjed makeup, facial hair and age related aturibute directions leave basic mouth shape/smile unchanged In process of running more of these studies on complete set of atuributes
P • 39
Generative techniques can be used to detect unintended and undesirable bias in facial analysis Equalizing error statistics across difgerent groups (defjned along cultural, demographic, phenotypical lines) is imporuant but not suffjcient for building fair, equitable, just or inclusive technology This analysis should be paru of a larger, socially contextualized, project to critically assess broader ethical concerns relating to facial analysis technology
P • 40
○ i.e. mine GANs for data, not people
P • 41
Countergactual fairness
Kilberuus et al. Avoiding discrimination through causal reasoning. NIPS, 2017. Kusner et al. Countergactual fairness. NIPS, 2017.
Countergactual fairness for text
Garg et al. Countergactual Fairness in Text Classifjcation through Robustness. AIES, 2019
Individual fairness
Dwork et al. Fairness Through Awareness. ITCS, 2012.
Model interpretability
Kim et al. Interpretability beyond feature aturibution: Quantitative testing with concept activation vectors (tcav). ICML, 2018. Chang et al. Explaining image classifjers by countergactual generation. ICLR, 2019. Fong and Vedaldi. Interpretable explanations of black boxes by meaningful peruurbation. ICCV, 2017. Dabkowski and Gal. Real time image saliency for black box classifjers. NIPS, 2017 Simonyan et al. Deep inside convolutional networks: Visualising image classifjcation models and saliency maps. 2013
P • 42
Denton et al. Detecting Bias with Generative Countergactual Face Aturibute Augmentation. CVPR Workshop on Fairness, Accountability, Transparency and Ethics in Computer Vision, 2019.