6.15.2020 ISIC Skin Image Analysis Workshop @ CVPR 2020 Project - - PowerPoint PPT Presentation

▶
6 15 2020
SMART_READER_LITE
LIVE PREVIEW

6.15.2020 ISIC Skin Image Analysis Workshop @ CVPR 2020 Project - - PowerPoint PPT Presentation

Dec 25, 2022 145 likes •334 views

Agreement Between Saliency Maps and Human-Labeled Regions of Interest Applications to Skin Disease Classifjcation Nalini Singh , Kang Li, David Coz, Christof Angermueller, Aaron Loh, Susan Huang, Yuan Liu 6.15.2020 ISIC Skin Image Analysis

slide-1
SLIDE 1

Agreement Between Saliency Maps and Human-Labeled Regions of Interest

Applications to Skin Disease Classifjcation

6.15.2020

Nalini Singh, Kang Li, David Coz, Christof Angermueller, Aaron Loh, Susan Huang, Yuan Liu

slide-2
SLIDE 2

Project Overview

Determine if a skin disease classifjcation model makes decisions for surprising reasons Quantify agreement between model explanations and human-labeled regions of interest

Goal Approach

ISIC Skin Image Analysis Workshop @ CVPR 2020

slide-3
SLIDE 3

Experiment Pipeline

Thresholded Dice Score Spearman Rank Correlation Overlay Input Image Majority Consensus ROI Labeling by 3 Human Graders Saliency Map Saliency Segmentation

ISIC Skin Image Analysis Workshop @ CVPR 2020

slide-4
SLIDE 4

Experiment Pipeline

Input Image

ISIC Skin Image Analysis Workshop @ CVPR 2020

  • 19,870 de-identifjed adult dermatology cases
  • 1-6 consumer-grade camera images + metadata per case
  • Classes: 26 skin conditions + 'Other'
  • Labels from aggregated board-ceruifjed dermatologist opinions

Model Development Dataset*

*Liu, Y., Jain, A., Eng, C. et al. A deep learning system for difgerential diagnosis of skin diseases. Nat Med (2020).

  • 1,309 de-identifjed adult dermatology cases sampled at random

from model development test set

Saliency Evaluation Dataset

slide-5
SLIDE 5

Experiment Pipeline

Overlay Input Image Majority Consensus ROI Labeling by 3 Human Graders

ISIC Skin Image Analysis Workshop @ CVPR 2020

slide-6
SLIDE 6

Experiment Pipeline

Input Image Saliency Map

ISIC Skin Image Analysis Workshop @ CVPR 2020

Model Architecture*

Clinical Images (1~6) Age: 31 Have fever?: No

..

Input image Concat Average Feature transform Input metadata Metadata (45) Acne Eczema Psoriasis Other Tinea Cyst Alopecia Areata Melanoma

..

Inception-v4 ..

27 Classes

Inception-v4

Input image Input metadata Feature transform Softmax *Liu, Y., Jain, A., Eng, C. et al. A deep learning system for difgerential diagnosis of skin diseases. Nat Med (2020).

slide-7
SLIDE 7

Experiment Pipeline

Input Image Saliency Map

ISIC Skin Image Analysis Workshop @ CVPR 2020

  • Top-1 accuracy: 66%

Model Architecture

*Sundararajan, Mukund, Ankur Taly, and Qiqi Yan. "Axiomatic aturibution for deep networks." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.

  • Integrated Gradients:

Saliency Map Generation*

slide-8
SLIDE 8

Experiment Pipeline

Input Image Saliency Map Saliency Segmentation

ISIC Skin Image Analysis Workshop @ CVPR 2020

slide-9
SLIDE 9

Experiment Pipeline

Thresholded Dice Score Spearman Rank Correlation Overlay Input Image Majority Consensus ROI Labeling by 3 Human Graders Saliency Map Saliency Segmentation

ISIC Skin Image Analysis Workshop @ CVPR 2020

slide-10
SLIDE 10

maximum maximum

Experiment Pipeline

Image Thresholded Dice Score Image Spearman Rank Correlation Input Case

ISIC Skin Image Analysis Workshop @ CVPR 2020

Case Spearman Rank Correlation Case Thresholded Dice Score

slide-11
SLIDE 11

Examples: High Agreement

ISIC Skin Image Analysis Workshop @ CVPR 2020

Incorrectly Classified

8 8 8 8 8 8 8 8 8 8 8 8 8 8 8

Correctly Classifjed Incorrectly Classifjed

slide-12
SLIDE 12

Examples: Low Agreement

ISIC Skin Image Analysis Workshop @ CVPR 2020

Correctly Classifjed Incorrectly Classified

8 8 8 8 8 8

Incorrectly Classifjed

slide-13
SLIDE 13

Results by Condition

ISIC Skin Image Analysis Workshop @ CVPR 2020

Dice Score Spearman Rank Correlation

slide-14
SLIDE 14

Androgenetic Alopecia (5)

8 8 8

Acne (1)

8 8 8

ISIC Skin Image Analysis Workshop @ CVPR 2020

Results by Condition

Dice Score Spearman Rank Correlation

slide-15
SLIDE 15

ISIC Skin Image Analysis Workshop @ CVPR 2020

Results by Condition

Melanoma (12) SK/ISK (18) Dice Score Spearman Rank Correlation

slide-16
SLIDE 16

Results by Demographics

ISIC Skin Image Analysis Workshop @ CVPR 2020

slide-17
SLIDE 17

Quantitatively compared model explanations to human-labeled ROIs:

  • Notably, found that model explanations point to 'normal anatomy' (e.g.

hair, nails, and lips).

  • Insights from analysis will guide targeted data collection and data

augmentation strategies.

  • Workfmow could be used to identify difgerences between model

explanations and human regions of interest for any model.

Summary & Conclusions

ISIC Skin Image Analysis Workshop @ CVPR 2020

slide-18
SLIDE 18
  • Eng, Clara, Y. Liu, and R. Bhatnagar. "Measuring clinician–machine

agreement in difgerential diagnoses for dermatology." British Journal of Dermatology (2019).

  • Liu, Yuan, et al. "A deep learning system for difgerential diagnosis of skin

diseases." Nature Medicine (2020): 1-9.

  • Ghorbani, Amirata, et al. "DermGAN: Synthetic Generation of Clinical Skin

Images with Pathology." NeurIPS ML4H Workshop (2019).

  • Singh, Nalini, et al., “Agreement Between Saliency Maps and

Human-Labeled Regions of Interest: Applications to Skin Disease Classifjcation.”, CVPR ISIC Workshop (2020).

Related Work

ISIC Skin Image Analysis Workshop @ CVPR 2020