Angular Visual Hardness Beidi Chen Department of Computer Science, - - PowerPoint PPT Presentation

Angular Visual Hardness

Beidi Chen

Department of Computer Science, Rice University Collaborators: Weiyang Liu, Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan kautz, and Anima Anandkumar

PART

Motivation

Applications Discoveries Background Motivation Conclusion

Human Visual Hardness

Image Degradation Semantic Ambiguity

Applications Discoveries Background Motivation Conclusion

Gap between human visual system and CNNs

Nail 0.93 0.2 Oil Filter 0.998 0.2 Easy for Human and Hard for CNNs Hard for Human and Easy for CNNs

C l a s s N a m e Softmax Score H u m a n S e l e c t i

• n

F r e q u e n c y

Golf Ball 0.001 1.0 Radio 0.001 1.0

Agenda

Part 1

Background

Part 2

Discoveries

Part 3 Part 4

Applications Conclusion

01

PART

Background

Applications Discoveries Background Motivation Conclusion

Do ImageNet Classifiers Generalize to ImageNet?

Human Labeling Interface

Recht et al. “Do ImageNet Classifiers Generalize to ImageNet?” ICML 2019

Applications Discoveries Background Motivation Conclusion

Loss function of CNNs in visual recognition

Softmax cross-entropy loss

The angle between feature and classifier The magnitude information Model Confidence

02

PART

Discoveries

Applications Discoveries Background Motivation Conclusion Given a sample x with label y: where,

1 2 3 4 5 6 7 8 9

Norm ||x|| Angle θ(x,wy) Classifier wy

wi is the classifier for the i-th class.

Proposal: Angular Visual Hardness (AVH)

Theoretical Foundation: Soudry et al. “The Implicit Bias of Gradient Descent on Separable Data” ICLR 2018

Applications Discoveries Background Motivation Conclusion

Simple Example: AVH vs. ||x|| ||x||

Raw data Color map of AVH Color map of ||x|| ||x||

Applications Discoveries Background Motivation Conclusion

CNN characteristics vs. human selection frequency

Applications Discoveries Background Motivation Conclusion

AVH is well aligned with human frequency

Spearman rank correlations

Applications Discoveries Background Motivation Conclusion

Discovery 1

Alexnet VGG19 ResNet50 Alexnet VGG19 Resnet50

AVH hits a plateau very early even when the accuracy or loss is still improving

Applications Discoveries Background Motivation Conclusion

Alexnet VGG19 ResNet50 Alexnet VGG19 Resnet50

Discovery 2

AVH is an indicator of model’s generalization ability

Applications Discoveries Background Motivation Conclusion

Alexnet VGG19 Resnet50 Alexnet VGG19 ResNet50

Discovery 3

The norm of feature embeddings keeps increasing during training

Applications Discoveries Background Motivation Conclusion

Discovery 4

AVH’s correlation with human selection frequency holds across models throughout training

Alexnet VGG19 ResNet50 Alexnet VGG19 Resnet50

Applications Discoveries Background Motivation Conclusion

Discovery 5

The norm’s correlation with human selection frequency is not consistent

Alexnet VGG19 ResNet50 Alexnet VGG19 Resnet50

Applications Discoveries Background Motivation Conclusion

Conjecture on training dynamic of CNNs

• Softmax cross-entropy loss will first optimize the angles among

different classes while the norm will fluctuate and increase very slowly.

• The angles become more stable and change very slowly while the norm

increases rapidly.

• Easy examples: the angles get decreased enough for correct

classification, the softmax cross-entropy loss can be well minimized by increasing the norm.

• Hard examples: the plateau is cause by unable to decrease the angle to

correctly classify examples or increase the norms otherwise hurting loss.

03

PART

Applications

Applications Discoveries Background Motivation Conclusion

Self-training and Domain Adaptation

road sidewalk building wall fence pole traffic lgt traffic sgn vegetation terrain sky person rider car truck bus train motorcycle bike

Zou et al. “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training” ECCV

Applications Discoveries Background Motivation Conclusion

AVH for Self-training and Domain Adaptation

Replace Softmax-based confidence with AVH-based one during sample selection: Similarly, AVH-based pseudo label

Applications Discoveries Background Motivation Conclusion

Main Results

Applications Discoveries Background Motivation Conclusion

Inner Metric

Examples chosen by AVH but not Softmax

Applications Discoveries Background Motivation Conclusion

AVH-based loss for Domain Generalization

AVH-based Loss:

04

PART

Conclusion

Applications Discoveries Background Motivation Conclusion

Summary

• Propose AVH as a measure for visual hardness
• Validate that AVH has a statistically significant stronger correlation with

human selection frequency

• Make observations on the dynamic evolution of AVH scores during

ImageNet training

• Show the superiority of AVH with its application to self-training for

unsupervised domain adaptation and domain generalization

Applications Discoveries Background Motivation Conclusion

Trajectory

Discussions

Trajectory of an adversarial example switching from one class to another

• Connection to deep metric learning
• Connection to fairness in machine learning
• Connection to knowledge transfer and

curriculum learning

• Uncertainty estimation (Aleatoric and

Epistemic)

• Adversarial Example: A Counter Example?

THANKS

Paper URL Contact: beidi.chen@rice.edu