Debiasing Skin Lesion Datasets and Models? Not So Fast Alceu - - PowerPoint PPT Presentation

▶
debiasing skin lesion datasets and models not so fast
SMART_READER_LITE
LIVE PREVIEW

Debiasing Skin Lesion Datasets and Models? Not So Fast Alceu - - PowerPoint PPT Presentation

Feb 25, 2023 43 likes •283 views

Debiasing Skin Lesion Datasets and Models? Not So Fast Alceu Bissoto, Eduardo Valle, Sandra Avila RECOD Lab., IC, University of Campinas (UNICAMP), Brazil RECOD Lab., DCA, FEEC, University of Campinas (UNICAMP), Brazil ISIC Workshop @

slide-1
SLIDE 1

Debiasing Skin Lesion Datasets and Models? Not So Fast

Alceu Bissoto¹, Eduardo Valle², Sandra Avila¹

¹RECOD Lab., IC, University of Campinas (UNICAMP), Brazil ²RECOD Lab., DCA, FEEC, University of Campinas (UNICAMP), Brazil

ISIC Workshop @ CVPR 2020

slide-2
SLIDE 2

2

Pigment Network Negative Network Globules Milia-like cysts Streaks

https://dermoscopedia.org/

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Medical Criteria

Asymmetry Border Regularity Color Diameter

slide-3
SLIDE 3

(De)Constructing Bias in Skin Lesion Datasets, Bissoto et al., ISIC Workshop @ CVPR 2019

3

However, previously on ...

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-4
SLIDE 4

4

Objective

Annotation regarding 7 visual artifacts that can lead to dataset biases How those artifacts affect classification models? Bias removal in the skin lesion context

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-5
SLIDE 5

4

Objective

Annotation regarding 7 visual artifacts that can lead to dataset biases How those artifacts affect classification models? Bias removal in the skin lesion context

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-6
SLIDE 6

4

Objective

Annotation regarding 7 visual artifacts that can lead to dataset biases How those artifacts affect classification models? Bias removal in the skin lesion context

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-7
SLIDE 7

Custom Data

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-8
SLIDE 8

Suspect Artifacts

6

Dark Corners (Vignetting) Hair Gel Border Patches Ink Markings

Ruler

Gel Bubble

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-9
SLIDE 9

7

Spearman Correlation w.r.t. diagnosis

Dark Corners

0.08

Hair

  • 0.08

Gel Border

  • 0.10

Gel Bubble

0.01

Ruler

0.10

Ink Markings

  • 0.07

Patches

  • 0.13

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Suspect Artifacts

slide-10
SLIDE 10

Spearman Correlation w.r.t. diagnosis Models’ Identification Performance

Dark Corners

95.6% 0.08

Hair

94.0%

  • 0.08

Gel Border

93.4%

  • 0.10

85.3%

Gel Bubble

0.01

Ruler

98.2% 0.10

Ink Markings

97.8%

  • 0.07

Patches

98.2%

  • 0.13

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Suspect Artifacts

7

slide-11
SLIDE 11

8

Dataset Traditional (%) Skin Only (%) Bbox (%) Bbox70 (%) ISIC 86.3 77.3 77.1 71.1 ISIC Normalized 81.5 72.7 67.0 59.8 Cross-dataset 83.5 72.3 71.3 71.5 Cross-dataset Normalized 77.1 69.0 67.2 64.1

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Normalized Datasets

slide-12
SLIDE 12

8

Dataset Traditional (%) Skin Only (%) Bbox (%) Bbox70 (%) ISIC 86.3 77.3 77.1 71.1 ISIC Normalized 81.5 72.7 67.0 59.8 Cross-dataset 83.5 72.3 71.3 71.5 Cross-dataset Normalized 77.1 69.0 67.2 64.1

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Normalized Datasets

slide-13
SLIDE 13

9

Trap Sets

Uses artifacts to purposefully mislead classifiers. Non-random splits maximize artifact bias on train and opposite bias

  • n test.

Models that ignore artifacts should be unaffected. Models that exploit biased should fail catastrophically (all tested models did).

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-14
SLIDE 14

10

Trap Sets

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Train

Spearman Correlation

Dark Corners

0.30

Hair

  • 0.26

Gel Border

0.12

Gel Bubble

  • 0.18

Ruler

0.41

Ink Markings

0.21

Patches

  • 0.11

Test

Spearman Correlation

  • 0.39

0.34

  • 0.51

0.47

  • 0.67
  • 0.42
  • 0.16
slide-15
SLIDE 15

11

Traditional Normalized Bbox Bbox

What features are being used?

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

slide-16
SLIDE 16

11

Traditional Normalized Bbox Bbox

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

What features are being used?

slide-17
SLIDE 17

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Debiasing Experiments

slide-18
SLIDE 18

13

Debiasing - Learning not to Learn (LNTL)

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Feature Extractor (ResNet18 / ResNet152)

Lesion Diagnosis Hair Classifier Dark Corner Classifier Patches Classifier

...

Benign Malignant Present Absent Present Absent Present Absent Learning not to Learn: Training Deep Neural Networks with Biased Data, Kim et al., CVPR 2019

slide-19
SLIDE 19

13

Debiasing - Learning not to Learn (LNTL)

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Feature Extractor (ResNet18 / ResNet152)

Lesion Diagnosis Hair Classifier Dark Corner Classifier Patches Classifier

...

Benign Malignant Present Absent Present Absent Present Absent Learning not to Learn: Training Deep Neural Networks with Biased Data, Kim et al., CVPR 2019

slide-20
SLIDE 20

14

Experiment Architecture Trap Test (%) Atlas Dermato (%) Atlas Clinical (%) Unchanged Inceptionv4 52.6 78.5 63.4 Normalized Inceptionv4 55.8 72.4

  • LNTL

ResNet152 54.5 78.4 70.1 Unchanged ResNet18 44.7 72.2 65.8 Normalized ResNet18 62.4 70.5

  • LNTL

ResNet18 51.4 76.0 68.2

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Debiasing

slide-21
SLIDE 21

Traditional models are less biased than previously thought (but they are still biased). Debiasing methods struggle to deal with the skin cancer. Domain adaptation, representation learning and disentanglement for more robust classifiers.

15 Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Conclusions

slide-22
SLIDE 22

Traditional models are less biased than previously thought (but they are still biased). Debiasing methods struggle to deal with the skin cancer. Domain adaptation, representation learning and disentanglement for more robust classifiers.

15 Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Conclusions

slide-23
SLIDE 23

Traditional models are less biased than previously thought (but they are still biased). Debiasing methods struggle to deal with the skin cancer. Domain adaptation, representation learning and disentanglement for more robust classifiers.

15 Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Conclusions

slide-24
SLIDE 24

Thank you!

Debiasing Skin Lesion Datasets and Models? Not So Fast, Bissoto et al., ISIC Workshop @ CVPR 2020

Alceu Bissoto alceubissoto@ic.unicamp.br Eduardo Valle dovalle@dca.fee.unicamp.br Sandra Avila @sandraavilabr

Code & Data: