Breaking Inter-Layer Co-Adaptation by Classifier Anonymization Ikuro - - PowerPoint PPT Presentation
ICML2019 Breaking Inter-Layer Co-Adaptation by Classifier Anonymization Ikuro Sato 1 Denso IT Laboratory. Inc., Japan 1 Kohta Ishikawa 1 National Institute of Advanced Industrial 2 Guoqing Liu 1 Science and Technology, Japan Masayuki Tanaka 2
/10
1
Denso IT Laboratory. Inc., Japan National Institute of Advanced Industrial Science and Technology, Japan
1 2
ICML2019
/10
2
/10
3
πβ, π β = arg min
π,π
1 π 0 ΰ·
π¦,π’ βπ
π π·π πΊπ π¦ , π’ π¦ πΊπ π¦ π·π πΊπ π¦ π π·π πΊπ π¦ , π’
Feature Ext. Classifier Input Loss w/ target π’ DNN
Feature dim-1 Feature dim-2 color: π·π value β+1β β-1β
Toy ex.) 2-class regression Features may form excessively complex distribution.
Feature extractor πΊπβ adapts to a particular classifier π·π. E2E opt.
/10
4
πβ = arg min
π
1 π 0 ΰ·
π¦,π’ βπ
π½π~Ξππ π·π πΊπ π¦ , π’
FOCA Random weak classifier: π~Ξπ
Want to know more about πͺπ? Please come to the poster!
Features form simple point-like distribution per class under some conditions.
Feature extractor πΊπβ adapts to a set of weak classifiers π·π .
Feature dim-1 Feature dim-2
/10
5
Please see the paper for the proof.
In words, If feature extractor has an enough representation ability, all input data of the same class are projected to a single point in the feature space in a class-separable way under certain conditions.
/10
6
x-axis y-axis Feature dim. #1 Feature dim. #2 data used to generate classifier decision boundary
start end
Small perturbations lead to point-like distribution. Small-batch classifier works as a weak classifier to the entire dataset.
/10
7
Thing we wish to confirm:
full-dataset classifier partial-dataset classifier
Do they perform similarly for given πΊπβ ??
/10
8
classifier trained with large dataset classifier trained with small dataset CIFAR10 test error rates Performance gap much smaller for FOCA large for
One indication of point-like property
(The same, fixed feature extractor is used within each method.)
/10
9
including:
between large- and small-dataset solutions
to further study the point-like property.
/10 10
What? How? Theory? Reality? Breaking co-adaptation between feature extractor and classifier. By classifier anonymization. Point-like property largely confirmed on real datasets. Proved: Features form simple point-like distribution.