Computational single-cell classification using deep learning on - - PowerPoint PPT Presentation

▶
computational single cell classification using deep
SMART_READER_LITE
LIVE PREVIEW

Computational single-cell classification using deep learning on - - PowerPoint PPT Presentation

Jul 08, 2023 330 likes •574 views

Computational single-cell classification using deep learning on bright-field and phase images Nan Meng, Hayden K.-H. So, Edmund Y. Lam Imaging Systems Laboratory, Department of Electrical and Electronic Engineering, University of Hong Kong

slide-1
SLIDE 1

Computational single-cell classification using deep learning on bright-field and phase images

Nan Meng, Hayden K.-H. So, Edmund Y. Lam

Imaging Systems Laboratory, Department of Electrical and Electronic Engineering, University of Hong Kong http://www.eee.hku.hk/isl

15th IAPR International Conference on Machine Vision Applications

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 1 / 23

slide-2
SLIDE 2

In a nutshell

MCF7 OAC OST PBMC THP1

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 2 / 23

slide-3
SLIDE 3

Introduction

1

Introduction

2

Network Design

3

Channel Augmentation

4

Results

5

Conclusions and Future Work

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 3 / 23

slide-4
SLIDE 4

Introduction

Ultrafast imaging

Enabling technology #1: Time-stretch imaging Asymmetric-detection time-stretch optical microscopy (ATOM) for

  • btaining label-free, high-contrast image of the transparent cells at

ultrahigh speed, and with sub-cellular resolution.

Figure: Photo of an ATOM system.

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 4 / 23

slide-5
SLIDE 5

Introduction

Ultrafast imaging

Figure: General schematic of an ATOM system.

Top speed: ≈ 30, 000 images per second Original image resolution: 84 × 305 Pixel superresolution gives: 305 × 305 Four bright-field images captured concurrently Data rate: 30000 × 84 × 305 × 4 ≈ 3.1 GB/s Detection mechanism: line by line via XXXXXX Cell flow: optofluidic system

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 5 / 23

slide-6
SLIDE 6

Introduction

Cell classification

Enabling technology #2: Deep learning for image classification Cell classification (phenotype): identify some specific cells among many different cells based on image analysis techniques. Data-driven methods for object classification. Automatically extract features to identify different types of cells.

Feature detector

cell images Features

Hypothesis formation

Candidate

  • bjects

Hypothesis verification Modelbase

class Objects

Deep Learning Algorithm CONV POOL CONV POOL FC SOFTMAX

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 6 / 23

slide-7
SLIDE 7

Introduction

Introduction

Deep learning breaks the desired complicated mapping into a series of nested simple mappings, each described by a different layer of the model.

Visible layer (input pixels) 1st hidden later (edges) 2nd hidden layer (corners and contours) 3rd hidden layer (object parts) Output (object identity)

Figure: Deep learning framework.

The input is presented at the visible layer. A series of hidden layers extracts increasingly abstract features from the image. Finally, the last layer learns descriptions of the image in terms of the object parts and use them to recognize objects.

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 7 / 23

slide-8
SLIDE 8

Network Design

1

Introduction

2

Network Design

3

Channel Augmentation

4

Results

5

Conclusions and Future Work

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 8 / 23

slide-9
SLIDE 9

Network Design

Network Design

Convolutional Neural Network (CNN) We explore a systematic way to design the network and tune the structure to obtain a robust model to avoid overfitting.

Block Convolutional Layer 1 Batch Norm 1 ReLU1

100 100 32 64 128 24 24 ReLU 2 Convolutional Layer 2 Convolutional Layer 3 ReLU 3 Batch Norm 3 12 12 1 1 3 6 305 305 11 5 Input images Fully connected layer 1 64 1 5 Fully connected layer 2 Batch Norm 2

Figure: Our proposed CNN-based framework.

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 9 / 23

slide-10
SLIDE 10

Network Design

Network Design

Building blocks Convolution layer extracts robust features for translation and rotation variations. Pooling layer makes the output less redundant. Batch normalization layer is effective to avoid overfitting.

Block Convolutional Layer 1 Batch Norm 1 ReLU1

305 305 11 5

Input Images translation invariance reduce redundancy avoid

  • verfitting

Combine these three connected layers as a basic feature extraction unit, what we call a “block”. Cascade multiple blocks to get the final framework.

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 10 / 23

slide-11
SLIDE 11

Network Design

Network Design

Cascading the blocks

1 Deep learning models can extract high-level features. 2 Higher layers are more likely to extract abstract and invariant features.

Block Convolutional Layer 1 Batch Norm 1 ReLU1

100 100 32 64 128 24 24 ReLU 2 Convolutional Layer 2 Convolutional Layer 3 ReLU 3 Batch Norm 3 12 12 1 1 3 6 305 305 11 5 Input images Fully connected layer 1 64 1 5 Fully connected layer 2 Batch Norm 2

block 1 activations block 2 activations input classifier

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 11 / 23

slide-12
SLIDE 12

Channel Augmentation

1

Introduction

2

Network Design

3

Channel Augmentation

4

Results

5

Conclusions and Future Work

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 12 / 23

slide-13
SLIDE 13

Channel Augmentation

Channel Augmentation

Bright-field and phase images Bright-field imaging is a technique where light from the specimen and its surroundings is collected to form an image against a bright background.

PCIe/Infiniband Input from host CPU I1 Pixel Stream I1 I2 I3 I4 I2 Pixel Stream I3 Pixel Stream I4 Pixel Stream

Frequency Domain Module

phase Figure: System Architecture.

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 13 / 23

slide-14
SLIDE 14

Channel Augmentation

Channel Augmentation

Computation of phase image φ(x, y): φ(x, y) = Im

  • F−1
  • C,

κx = κy = 0

F{G(x,y)·FOV} 2π·(κx+iκy) ,

  • therwise
  • (1)

∇φx and ∇φy : local phase shift G(x, y) = ∇φx + i · ∇φy F and F−1 : Forward and inverse Fourier transforms (κx + iκy) : Fourier spatial frequencies normalized as a linear ramp C : An arbitrary integration constant FOV : image field of view expressed in physical units

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 14 / 23

slide-15
SLIDE 15

Channel Augmentation

Channel Augmentation

Generating the phase image aims to enrich information of each individual sample without increasing the size of the dataset used for training.

[305 x 305] [305 x 305] [305 x 305] [305 x 305] [305 x 305] [305 x 305 x 5]

CNN-based framework

Output classes

Figure: Channel augmentation cascades several relevant images together to enrich information.

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 15 / 23

slide-16
SLIDE 16

Results

1

Introduction

2

Network Design

3

Channel Augmentation

4

Results

5

Conclusions and Future Work

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 16 / 23

slide-17
SLIDE 17

Results

Results

Channel Augmentation is efficient for cell classification problem. Different channel images provide competitive information. Channel images contain better features than phase image. Cascading channel and phase images achieves best classification.

Aspects Average Accuracy validation test channel 1 0.94 0.94 channel 2 0.96 0.95 channel 3 0.94 0.96 channel 4 0.94 0.92 channel 1-4 0.96 0.93 phase only 0.93 0.90 channel 1-4 & phase 0.97 0.97

Table: Classification accuracy with different channel augmentation strategies.

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 17 / 23

slide-18
SLIDE 18

Results

Results

The Precision of each class of testset Different channels perform discriminatively on specific type of cells. Model preforms best after channel augmentation. precision = tp tp + fp tp : true positive fp : false positive

Table: Precision of test

Aspects Precision of each class test THP1 OAC MCF7 PBMC OST channel 1 0.8429 0.9551 1.000 0.8795 1.000 channel 2 0.9296 0.9872 0.9868 0.8588 0.9889 channel 3 0.9444 0.9753 1.000 0.9255 0.9753 channel 4 0.7838 0.9773 0.9880 0.8462 0.9870 channel 1-4 1.000 0.9859 0.9767 0.7907 0.9259 phase 0.8571 0.9167 0.9870 0.8642 0.8642 channel 1-4 &phase 1.000 1.000 0.9524 0.9351 0.9324

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 18 / 23

slide-19
SLIDE 19

Results

Results

The Recall of each class of testset Different channels perform discriminatively on specific type of cells. Model preforms best after channel augmentation. recall = tp tp + fn tp : true positive fn : false negative

Table: Recall of test

Aspects Recall of each class test THP1 OAC MCF7 PBMC OST channel 1 0.8551 1.000 0.9634 0.9012 0.9518 channel 2 0.8354 0.9872 1.000 0.9359 0.9889 channel 3 0.8947 0.9875 1.000 0.9355 1.000 channel 4 0.7945 1.000 1.000 0.8048 0.9870 channel 1-4 0.9870 0.9859 0.9882 0.9189 0.8065 phase 0.8571 0.8652 1.000 0.8642 0.9091 channel 1-4 &phase 0.9756 0.9770 1.000 0.9351 0.9324

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 19 / 23

slide-20
SLIDE 20

Results

Results

The F1 score of each class of testset. Different channels perform discriminatively on specific type of cells. Model preforms best after channel augmentation. F1 = 2× precision · recall precision + recall

Table: F1 score of test

Aspects F1 score of each class test THP1 OAC MCF7 PBMC OST channel 1 0.85 0.98 0.98 0.89 0.98 channel 2 0.88 0.99 0.99 0.90 0.99 channel 3 0.92 0.98 1.0 0.93 0.99 channel 4 0.79 0.99 0.99 0.83 0.99 channel 1-4 0.99 0.98 0.98 0.85 0.86 phase 0.87 0.90 0.99 0.88 0.89 channel 1-4 &phase 0.99 1.0 0.99 0.95 0.94

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 20 / 23

slide-21
SLIDE 21

Conclusions and Future Work

1

Introduction

2

Network Design

3

Channel Augmentation

4

Results

5

Conclusions and Future Work

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 21 / 23

slide-22
SLIDE 22

Conclusions and Future Work

Conclusions and Future Work

Some conclusions: CNN-based models after careful design can have strong capacities to learn representational features, which further improve the classification accuracy Channel augmentation strategy via cascading several relevant images is effective for the network to learn good representations in cell classification task Future work: Investigate further the abilities of CNN-based models to extract representational features Explore strategies for network design to extract resolution invariant features

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 22 / 23

slide-23
SLIDE 23

Conclusions and Future Work

Acknowledgment

This work was supported in part by National Natural Science Foundation of China (NSFC)/RGC under Hong Kong Research Grants Council (NHKU714/13) Hong Kong Research Grants Council General Research Fund (17245716) Croucher Innovation Award

  • E. Lam (University of Hong Kong)

15th IAPR-MVA 9 May 2017 23 / 23