Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach Proteek Roy and Vishnu Boddeti Michigan State University CVPR 2019 [~]$ [1/13]
>>> Representation Learning: The Bright Side * Deep Embeddings: E ( x , θ E ) z ∈ R d [~]$ [2/13]
>>> Representation Learning: The Bright Side * Deep Embeddings: E ( x , θ E ) z ∈ R d * Features contain a lot of information * basis for generalizing and transferring to other tasks [~]$ [2/13]
>>> Representation Learning: The Bright Side * Deep Embeddings: E ( x , θ E ) z ∈ R d * Features contain a lot of information * basis for generalizing and transferring to other tasks * Applications include: . . . . similarity . . best match R R . . . R . . . . . . R R Figure: Face Recognition Figure: Image Retrieval [~]$ [2/13]
High Resp. Low Resp. High Resp. Low Resp. Test Image Activations Neurons Gender Bangs Brown Hair Pale Skin Narrow Eyes High Cheek. (a.1) (a.2) Hair Color (b.1) (b.2) Eyeglasses Mustache Black Hair Smiling Big Nose (a.3) Age (a.4) Race (b.3) (a.5) Wear. Hat Blond Hair Wear. Lipstick Asian Big Eyes Face Shape (a.6) Eye Shape (a) ANet (FC) ANet (C4) ANet (C3) Identity-related Attributes Identity-non-related Attributes 100% 90% 95% 85% Accuracy 90% 80% 85% 75% 70% 80% Male White Black Asian Smiling Wearing Rosy 5oClock Hat Cheeks Shadow (b) ANet (After fine-tuning) HOG (After PCA) Average Accuracy 80% single best 70% performing 60% neuron 50% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Percentage of Best Performing Neurons Used >>> Representation Learning: The Dark Side [~]$ [3/13]
High Resp. Low Resp. High Resp. Low Resp. Test Image Activations Neurons Gender Bangs Brown Hair Pale Skin Narrow Eyes High Cheek. (a.1) (a.2) Hair Color (b.1) (b.2) Eyeglasses Mustache Black Hair Smiling Big Nose (a.3) Age (a.4) Race (b.3) (a.5) Wear. Hat Blond Hair Wear. Lipstick Asian Big Eyes Face Shape (a.6) Eye Shape (a) ANet (FC) ANet (C4) ANet (C3) Identity-related Attributes Identity-non-related Attributes 100% 90% 95% 85% Accuracy 90% 80% 85% 75% 70% 80% Male White Black Asian Smiling Wearing Rosy 5oClock Hat Cheeks Shadow (b) ANet (After fine-tuning) HOG (After PCA) Average Accuracy 80% single best 70% performing 60% neuron 50% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Percentage of Best Performing Neurons Used >>> Representation Learning: The Dark Side * Features contain a lot of information [~]$ [3/13]
High Resp. Low Resp. High Resp. Low Resp. Test Image Activations Neurons Bangs Brown Hair Pale Skin Narrow Eyes High Cheek. (a.1) Gender (a.2) Hair Color (b.1) (b.2) Eyeglasses Mustache Black Hair Smiling Big Nose (a.3) Age (a.4) Race (b.3) (a.5) Wear. Hat Blond Hair Wear. Lipstick Asian Big Eyes Face Shape (a.6) Eye Shape (a) ANet (FC) ANet (C4) ANet (C3) Identity-related Attributes Identity-non-related Attributes 100% 90% 95% 85% Accuracy 90% 80% 85% 75% 70% 80% Male White Black Asian Smiling Wearing Rosy 5oClock Hat Cheeks Shadow (b) ANet (After fine-tuning) HOG (After PCA) Average Accuracy 80% single best 70% performing 60% neuron 50% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Percentage of Best Performing Neurons Used >>> Representation Learning: The Dark Side * Features contain a lot of information * Information may inadvertently be sensitive [~]$ [3/13]
High Resp. Low Resp. High Resp. Low Resp. Test Image Activations Neurons Bangs Brown Hair Pale Skin Narrow Eyes High Cheek. (a.1) Gender (a.2) Hair Color (b.1) (b.2) Eyeglasses Mustache Black Hair Smiling Big Nose (a.3) Age (a.4) Race (b.3) (a.5) Wear. Hat Blond Hair Wear. Lipstick Asian Big Eyes Face Shape (a.6) Eye Shape (a) ANet (FC) ANet (C4) ANet (C3) Identity-related Attributes Identity-non-related Attributes 100% 90% 95% 85% Accuracy 90% 80% 85% 75% 70% 80% Male White Black Asian Smiling Wearing Rosy 5oClock Hat Cheeks Shadow (b) ANet (After fine-tuning) HOG (After PCA) Average Accuracy 80% single best 70% performing 60% neuron 50% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Percentage of Best Performing Neurons Used >>> Representation Learning: The Dark Side * Features contain a lot of information * Information may inadvertently be sensitive * compromise privacy of data owner * result in unfair or biased decision systems [~]$ [3/13]
>>> Representation Learning: The Dark Side * Features contain a lot of information * Information may inadvertently be sensitive * compromise privacy of data owner * result in unfair or biased decision systems * Soft attribute from face features * Reconstruction from face features High Resp. Low Resp. High Resp. Low Resp. Test Image Activations Neurons Bangs Brown Hair Pale Skin Narrow Eyes High Cheek. (a.1) Gender (a.2) Hair Color (b.1) (b.2) Eyeglasses Mustache Black Hair Smiling Big Nose (a.3) Age (a.4) Race 0.84 0.78 0.82 0.93 (b.3) (a.5) Wear. Hat Blond Hair Wear. Lipstick Asian Big Eyes Face Shape (a.6) Eye Shape Mai et al., PAMI 2018 Liu et al., ICCV 2015 (a) ANet (FC) ANet (C4) ANet (C3) Identity-related Attributes Identity-non-related Attributes 100% 90% [~]$ [3/13] 95% 85% Accuracy 90% 80% 85% 75% 70% 80% Male White Black Asian Smiling Wearing Rosy 5oClock Hat Cheeks Shadow (b) ANet (After fine-tuning) HOG (After PCA) Average Accuracy 80% single best 70% performing 60% neuron 50% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Percentage of Best Performing Neurons Used
>>> Central Aim of This Paper Mitigating Information Leakage Develop representation learning algorithms that can intentionally and permanently obscure sensitive information while retaining task dependent information. [~]$ [4/13]
>>> Problem Setting: Adversarial Representation Learning * Three player zero-sum game between: [~]$ [5/13]
>>> Problem Setting: Adversarial Representation Learning E ( x , θ E ) z ∈ R d * Three player zero-sum game between: * Encoder extracts features z [~]$ [5/13]
>>> Problem Setting: Adversarial Representation Learning E ( x , θ E ) z ∈ R d T ( x , θ T ) q T ( t | z ) * Three player zero-sum game between: * Encoder extracts features z * Target Predictor for desired task from features z [~]$ [5/13]
>>> Problem Setting: Adversarial Representation Learning E ( x , θ E ) z ∈ R d T ( x , θ T ) q T ( t | z ) A ( x , θ A ) q A ( s | z ) * Three player zero-sum game between: * Encoder extracts features z * Target Predictor for desired task from features z * Adversary extracts sensitive information from features z [~]$ [5/13]
>>> Problem Setting: Adversarial Representation Learning E ( x , θ E ) z ∈ R d T ( x , θ T ) q T ( t | z ) A ( x , θ A ) q A ( s | z ) * Three player zero-sum game between: * Encoder extracts features z * Target Predictor for desired task from features z * Adversary extracts sensitive information from features z * Minimum Likelihood Adversarial Representation Learning: θ E , θ T max min J 1 ( θ E , θ T ) − α J 2 ( θ E , θ A ) (1) θ A � �� � � �� � likelihood of predictor likelihood of adversary [~]$ [5/13]
>>> Optimizing Likelihood Can be Sub-Optimal [~]$ [6/13]
>>> Optimizing Likelihood Can be Sub-Optimal * Adversary 1.0 Probability 0.0 0.0 Sensitive Class [~]$ [6/13]
>>> Optimizing Likelihood Can be Sub-Optimal * Adversary * Encoder 1.0 Probability Probability 0.5 0.5 0.0 0.0 0.0 Sensitive Class Sensitive Class [~]$ [6/13]
>>> Optimizing Likelihood Can be Sub-Optimal * Adversary * Encoder * Equillibrium 1.0 Probability Probability Probability 0.5 0.5 0.33 0.33 0.33 0.0 0.0 0.0 Sensitive Class Sensitive Class Sensitive Class [~]$ [6/13]
>>> Optimizing Likelihood Can be Sub-Optimal * Adversary * Encoder * Equillibrium 1.0 Probability Probability Probability 0.5 0.5 0.33 0.33 0.33 0.0 0.0 0.0 Sensitive Class Sensitive Class Sensitive Class Limitations: * Encoder target distribution leaks information !! * Practice: simultaneous SGD does not reach equilibrium * Class Imbalance: likelihood biases solution to majority class [~]$ [6/13]
>>> Maximum Entropy Adversarial Representation Learning Key Idea Optimize the encoder to maximize entropy of adversary as opposed to minimizing its likelihood. [~]$ [7/13]
>>> Maximum Entropy Adversarial Representation Learning Key Idea Optimize the encoder to maximize entropy of adversary as opposed to minimizing its likelihood. * Adversary 1.0 Probability 0.0 0.0 Sensitive Class [~]$ [7/13]
>>> Maximum Entropy Adversarial Representation Learning Key Idea Optimize the encoder to maximize entropy of adversary as opposed to minimizing its likelihood. * Adversary * Encoder 1.0 Probability Probability 0.33 0.33 0.33 0.0 0.0 Sensitive Class Sensitive Class [~]$ [7/13]
Recommend
More recommend
