Lung Nodule Detection based on Convolutional Neural Networks Julio - - PowerPoint PPT Presentation

Lung Nodule Detection based on Convolutional Neural Networks

Julio Mendoza Bobadilla Advisor: Prof. Helio Pedrini

Master’s Defense Institute of Computing University of Campinas

December 18, 2017

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 1 / 43

Summary

• 1. Introduction

Problem Motivation Hypotheses Contributions

• 2. Background

CAD Systems Lung Area Segmentation Candidate Nodule Localization Candidate Nodule Classification

• 3. Methodology
• 4. Results
• 5. Conclusions

Future Work

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 2 / 43

Introduction

Problem

Input: Chest X-Ray Output: Potential nodules highlighted

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 3 / 43

Introduction

Motivation Estimated number of cancer deaths on both sexes (Source: World Cancer Report 20141).

Improve the performance of radiologists on lung cancer screening. Address the drawbacks of lung cancer screening with CT images. Explore the potential role of CAD systems with CXR for lung cancer screening. Leverage the advantages of deep learning approaches on lung nodule detection.

1McGuire, “World Cancer Report 2014. Geneva, Switzerland: World Health Organization, International Agency for Research on Cancer, WHO Press”, 2016. Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 4 / 43

Introduction

Hypotheses

Main hypothesis: Convolutional Neural Networks (CNNs) trained from the scratch can perform well on lung nodule classification by setting proper regularization and optimization methods. Secondary hypotheses: The design of a specialized architecture for the classification stage of lung nodule detection improve the performance of our method. The regularization effects of data augmentation, dropout and weight penalties critical for lung nodule classification due to the amount of samples used in training. The regularization and optimization effects of unsupervised objectives in loss functions improve CNN performance in lung nodule classification.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 5 / 43

Introduction

Contributions

Main contributions: An analysis and evaluation of methods for lung area segmentation and candidate nodule localization. The proposition of a method for lung nodule classification based a CNN trained from the scratch. The comparison of the our lung nodule detection pipeline with other approaches of the literature.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 6 / 43

Background

CAD Systems

Usage modalities: As first reader: radiologists analyze the regions detected by the CAD system. As second reader: CAD system may detect nodules missed by radiologists. CAD systems usually detect nodules solving three subproblems: Lung Area Segmentation Candidate Nodule Localization Candidate Nodule Classification

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 7 / 43

Background

Lung Area Segmentation

Lung Area Segmentation Candidate Nodule Localization Candidate Nodule Classification Lung Area Segmentation Candidate Nodule Localization Candidate Nodule Classification

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 8 / 43

Background

Active Appearance Model

Shape model

◮ Normalization with Procrustes Analysis. ◮ Finding bases with Principal Component Analysis.

s = ¯ s

• mean shape

+

n

• i=1

pi si

• shape bases

Appearance model

◮ Appearance representation using visual descriptors. ◮ Finding bases with Principal Component Analysis.

A = ¯ A

• mean appearance

+

m

• i=1

ci Ai

• appearance bases

Model instantiation M(W (x, p)) ≃ ¯ A +

m

• i=1

ciAi Model fitting p∗, c∗ = arg min

p,c

• M(W (x, p)) −
• ¯

A +

m

• i=1

ciAi

• 2

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 9 / 43

Background

Active Appearance Model Source: Matthews and Baker2

2Matthews and Baker, “Active Appearance Models Revisited”, 2004. Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 10 / 43

Background

Candidate Nodule Localization

Lung Area Segmentation Candidate Nodule Localization Candidate Nodule Classification

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 11 / 43

Background

Scale-Space based Detectors

t scale

L(·, ·; t) = g(·, ·; t) ∗ f (·, ·)

t scale

Laplacian of Gaussian local extrema blobs

Scale-space blob detection.

normalized Laplacian of Gaussian (LoG): t∇L = t(Lxx + Lyy) normalized Determinant of Hessian (DoH): t∇L = t2(LxxLyy − L2

xy)

Difference of Gaussians (DoG): DoG(t) = L(·, ·; t + ∆t) − L(·, ·; t)

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 12 / 43

Background

Convergence Index based detectors P Q

θ

(i, j) (k, l) gradient vector

R

Convergence degree at point Q: cos θ(k, l) Convergence index at point P with support region R: CI(i, j) = 1 M

• (k,l)∈R

cos θ(k, l)

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 13 / 43

Background

Sliding Band Filter P

d B0 B1 B15 rmin rmax+d B2 B3 ... ... B14

R

Convergence index at point P with support region R: SB(x, y) = 1 N

N−1

• i=0

      max

rmin≤n≤rmax

      1 d

n+d

• m=n

cos θi,m

• CI of a band in the line Bi

           

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 14 / 43

Background

Candidate Nodule Classification

Lung Area Segmentation Candidate Nodule Localization Candidate Nodule Classification

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 15 / 43

Background

Feature Engineering

Input nodules

. . .

Classifier Feature Extraction

Feature extraction: geometric, shape, intensity and gradient features. Binary classification.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 16 / 43

Background

Feature Engineering

Feature Extraction Input nodules

. .

Classifier Nodule Segmentation Feature Selection / Reduction

.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 17 / 43

Background

Convolutional Neural Networks

Convolutional Neural Networks

◮ Convolution ◮ Pooling

ConvNet Input nodules

. . .

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 18 / 43

Methodology

Overview Overview of the main steps of the proposed methodology.

We segment the lung area using patch-based AAM. We find candidate locations with an SB filter. We estimate the probability of nodules being candidates using a CNN.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 19 / 43

Methodology

Segmentation Overview of the segmentation stage.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 20 / 43

Methodology

Segmentation

Low resolution model High resolution model

Visualization of Patch-based AAM (for low and high resolution models) fitted on the sample JPCLN001.IMG from the JSRT dataset.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 21 / 43

Methodology

Detection

Input X-ray (JPCLN001.IMG) SB filter output Non-maxima suppression with rsbf = 7px

Candidate nodules detected with an SB filter.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 22 / 43

Methodology

Classification Overview of the classification stage.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 23 / 43

Methodology

Data Preparation and Augmentation

Data preparation: Labeling criterion: 25mm. Z-score normalization. Data augmentation: Affine transformations (translation, rotation, scaling, flip). Intensity shifts.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 24 / 43

Methodology

Architecture Design

• utput

... ...

k

3x3

2k

2c+1x 2c+1

ck

512

f layers c layers

Representation of the architecture ConvNet(c, k, f).

The network is composed of c alternating convolutional (in red), and max-pooling (in green) layers, followed by f fully connected layers (in yellow). The network is fed with images with size 2c+1 × 2c+1.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 25 / 43

Methodology

Learning

Algorithm 1 Stochastic Gradient Descent with balanced samples.

1: procedure SGD-BalancedSamples 2: Require: Network parameters θ 3: Require: Positive samples P 4: Require: Negative samples N 5:

while AUC does not stop improving do

6:

Select the first half of the minibatch from N iteratively.

7:

Select the second half of the minibatch from P randomly.

8:

Apply the data augmentation transformations to all samples of the minibatch.

9:

Compute gradient estimate with the minibatch.

10:

Update network parameters θ.

11:

end while

12: end procedure Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 26 / 43

Results

JSRT Dataset

Composed of 247 CXR images. 154 images with nodules and 93 without nodules. The annotations include nodule location, level of subtlety, effective diameter, malignancy condition, among

• thers.

Patches with nodules extracted from CXRs of the JSRT dataset.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 27 / 43

Results

SCR Dataset Left: The initial points marked by an observer on the first image of the JSRT database. Right: the interpolated landmarks along the contour for use of segmentation methods (Source: Van Ginneken, Stegmann, and Loog3).

Created to evaluate method for segmentation of lungs, hearth, and clavicles in CXR images. Composed of landmarks extracted from the images of the JSRT dataset.

3Van Ginneken, Stegmann, and Loog, “Segmentation of Anatomical Structures in Chest Radiographs using Supervised Methods: A Comparative Study

• n a Public Database”, 2006.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 28 / 43

Results

LIDC-IDRI Dataset

Contains 1018 CT scans, and 290 CXR associated with some of these CT scans. The annotations include nodule location, level of subtlety, effective diameter, malignancy condition, annotation confidence, among others. We filtered the annotation:

◮ Effective diameter ≥ 3mm ◮ Accepted by at least 3 radiologists.

We included nodules annotated by 2 radiologists when the sum of the confidence level ≥ 7 We excluded nodules that have an average subtlety level < 2.

Patches with nodules extracted from CXRs of the LIDC-IDRI dataset.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 29 / 43

Results

Segmentation Segmentation results on the positive subset of JSRT dataset. Segmentation results on the LIDC-IDRI dataset.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 30 / 43

Results

Segmentation Ω µ ± σ Min Max Level Set + Deep Belief Network4 0.985 ± 0.003 0.972 0.991 Non-Rigid Registration using Atlas5 0.954 ± 0.015

• Human Observer6

0.946 ± 0.018 0.822 0.972 Pixel Classification Post-Processed 6 0.945 ± 0.022 0.823 0.972 Proposed method (Patch-based AAM with DSIFT) 0.935 ± 0.019 0.786 0.964 ASM tunned 6 0.927 ± 0.032 0.745 0.964 Mean Shape 0.714 ± 0.075 0.461 0.889 Segmentation results in terms of the Jaccard coefficient Ω. Methods are ranked according to their mean.

4Ngo and Carneiro, “Fully Automated Segmentation Using Distance Regularised Level Set and Deep-Structured Learning and Inference”, 2017. 5Candemir et al., “Lung Segmentation in Chest Radiographs using Anatomical Atlases with Nonrigid Registration”, 2014. 6Van Ginneken, Stegmann, and Loog, “Segmentation of Anatomical Structures in Chest Radiographs using Supervised Methods: A Comparative Study

• n a Public Database”, 2006.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 31 / 43

Results

Detection

10 20 30 40 50 60 70 80 90 100 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

DoG, ABPI=87.46, AUC = 52.87 LoG, ABPI=174.81, AUC = 50.22 DoH, ABPI=118.05, AUC = 37.83 WMCI, ABPI=92.58, AUC = 47.43 SBF, ABPI=86.36, AUC = 53.75

FROC curves for detectors evaluated using LIDC-IDRI dataset.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 32 / 43

Results

Classification: Data Augmentation

1 2 3 4 5 6 7 8 9 10 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

No D.A., AUC = 4.56 D.A. F, AUC = 4.91 D.A. F+R, AUC = 5.49 D.A. F+R+T, AUC = 5.53 D.A. F+R+T+I, AUC = 5.61 D.A. F+R+T+I+S, AUC = 5.85

We evaluate the performance of various data augmentation schemes based on including transformations incrementally.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 33 / 43

Results

Classification: Dropout Schemes

1 2 3 4 5 6 7 8 9 10 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

No dropout, AUC = 5.85 Dropout 0.1, AUC = 5.71 Dropout 0.15, AUC = 5.85 Dropout 0.2, AUC = 5.86 Dropout 0.25, AUC = 5.99 Dropout 0.3, AUC = 5.71

constant dropout

1 2 3 4 5 6 7 8 9 10 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

No dropout, AUC = 5.85 Linear Inc. DP 0.075, AUC = 5.87 Linear Inc. DP 0.1, AUC = 5.93 Linear Inc. DP 0.125, AUC = 6.03 Linear Inc. DP 0.15, AUC = 6.02 Linear Inc. DP 0.175, AUC = 5.90

linear increasing dropout

Comparing constant dropout with linear increasing dropout on convolution layers.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 34 / 43

Results

Classification: Architecture Design

1 2 3 4 5 6 7 8 9 10 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

4 conv layer, AUC = 5.60 5 conv layer, AUC = 6.03 6 conv layer, AUC = 6.06

varying c with k = 64 and f = 1

1 2 3 4 5 6 7 8 9 10 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

Convnet(6, 16), AUC = 5.80 Convnet(6, 32), AUC = 6.15 Convnet(6, 48), AUC = 6.08 Convnet(6, 64), AUC = 6.06

varying k with c = 6 and f = 1

1 2 3 4 5 6 7 8 9 10 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

No FC layers, AUC = 6.04 1 FC layer, AUC = 6.15 2 FC layer, AUC = 5.97 3 FC layer, AUC = 6.01

varying f with c = 6 and k = 32

FROC curves varying the parameters c, k and f .

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 35 / 43

Results

Classification: Unsupervised Objectives

Only supervised objectives Supervised and unsupervised objectives

Experiments considering supervised and unsupervised objectives on small CNN (ConvNet(3, 32, 2)).

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 36 / 43

Results

Classification: Sources of Variation on Evaluation Protocols

CAD System Datasets Protocol Labeling Criteria Parameter Optimization Chen and Suzuki7 Private dataset, JSRT excluding 14 opaque cases Train on private dataset, test

• n JSRT subset

Distance (25 mm) Considers test results Hardie et al.8 Private dataset, JSRT excluding 14 opaque cases Train on a private dataset, test

• n JSRT, 10-fold

cross-validation on JSRT subset Distance (25 mm) Considers only training set Schilham et al.9 JSRT dataset 5-fold cross-validation on JSRT Overlap (> 0%) Considers test results Shiraishi et al.10 Private dataset augmented with JSRT images Train and test on the merged dataset Distance (22mm and 24mm) Considers only training set Coppini et al.11 Private dataset, JSRT subset of 140 samples 5 partitions of the JSRT subset Centroid Considers test results Wei et al.12 JSRT dataset Leave-one-out cross-validation Unspecified Considers test results Wang et al.13 JSRT excluding opaque cases 10-fold cross-validation Distance (unspecified) Not specified

Main sources of variation of the evaluation protocol used in competing methods.

7Chen and Suzuki, “Computerized Detection of Lung Nodules by Means of Virtual Dual-Energy Radiography”, 2013. 8Hardie et al., “Performance Analysis of a new Computer Aided Detection System for Identifying Lung Nodules on Chest Radiographs”, 2008. 9Schilham, Ginneken, and Loog, “A Computer-aided Diagnosis System for Detection of Lung Nodules in Chest Radiographs with an Evaluation on a Public Database”, 2006. 10Shiraishi et al., “Computer-aided Diagnostic Scheme for the Detection of Lung Nodules on Chest Radiographs: Localized Search Method based on Anatomical Classification”, 2006. 11Coppini et al., “Neural Networks for Computer-Aided Diagnosis: Detection of Lung Nodules in Chest Radiograms”, 2003. 12Wei et al., “Optimal Image Feature Set for Detecting Lung Nodules on Chest X-Ray Images”, 2002. 13Wang et al., “Lung Nodule Classification using Deep Feature Fusion in Chest Radiography”, 2017. Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 37 / 43

Results

Classification Considering Only Supervised Objetives CAD system performance comparison. Average Method Reported CNN FPPI Sensitivity (%) Sensitivity (%) 5.0 Chen and Suzuki14 a,c 85.0 90.0 5.0 Hardie et al.15 a 80.1 90.0 2.0 Schilham et al.16 51.0 71.4 4.0 Schilham et al. 16 71.0 79.2 5.0 Shiraishi et al.17 a,b 70.1 90.0 4.3 Coppini et al.18 60.0 79.9 5.4 Wei et al.19 80.0 82.5 1.19 Wang et al.20 a 69.3 72.1

aResults reported in this row exclude opaque cases.

bResults based on 924 chest radiographs that include the JRST cases. c A private database was used for training and the JSRT for testing. 14Chen and Suzuki, “Computerized Detection of Lung Nodules by Means of Virtual Dual-Energy Radiography”, 2013. 15Hardie et al., “Performance Analysis of a new Computer Aided Detection System for Identifying Lung Nodules on Chest Radiographs”, 2008. 16Schilham, Ginneken, and Loog, “A Computer-aided Diagnosis System for Detection of Lung Nodules in Chest Radiographs with an Evaluation on a Public Database”, 2006. 17Shiraishi et al., “Computer-aided Diagnostic Scheme for the Detection of Lung Nodules on Chest Radiographs: Localized Search Method based on Anatomical Classification”, 2006. 18Coppini et al., “Neural Networks for Computer-Aided Diagnosis: Detection of Lung Nodules in Chest Radiograms”, 2003. 19Wei et al., “Optimal Image Feature Set for Detecting Lung Nodules on Chest X-Ray Images”, 2002. 20Wang et al., “Lung Nodule Classification using Deep Feature Fusion in Chest Radiography”, 2017. Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 38 / 43

Results

Classification 1 2 3 4 5 6 7 8 9 10 Average FPs per Image 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Sensitivity

Convnet(6,32,1) Chen and Suzuki. Hardie et al. Schilham et al. Shiraishi et al. Coppini et al. Wei et al. Wang et al.

Comparison of the method performance on the JRST database. Sensitivity values are adjusted by considering opaque cases as missed.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 39 / 43

Results

Classification

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 40 / 43

Conclusions

Final Remarks

Our method aimed to detect potential location of nodules with high sensitivity through a few false positives per image.

◮ We analyzed and compared methods for lung area segmentation and candidate nodule

classification.

◮ We explored the effectiveness of CNNs for reducing false positives on the candidate

classification stage.

We showed that CNN trained from the scratch obtained good results on lung nodule classification.

◮ We analyzed the impact of various data augmentation transformations. ◮ We compared two ways to use dropout on convolutional layers. We found that

assigning increasing dropout probabilities converged faster and it was slightly superior than assigning fixed dropout.

◮ We evaluated the impact of CNN architecture parameters. We obtained best results

with ConvNet(6, 32, 1).

◮ We trained all models balancing batches on each SGD iteration. ◮ We showed that augmenting the loss function with an unsupervised term improved the

effectiveness of the methods.

We compared the results of the best configuration with competing approaches.

◮ A fair comparison is difficult due to variation on evaluation protocols of the methods. ◮ Under our considerations with respect to variations, we showed that our method

achieved good results when compared to the state-of-the-art.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 41 / 43

Conclusions

Future Work

Results obtained by considering supervised and unsupervised objectives on CNN training suggested that further investigation in this direction should be conducted. Results found in this dissertation can be leveraged to improve methods for lung nodule classification in CT images.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 42 / 43

Acknowledgements

IC/UNICAMP for the infrastructure. CAPES for the financial support.

Julio Mendoza Bobadilla IC/UNICAMP Lung Nodule Detection based on Convolutional Neural Networks December 18, 2017 43 / 43