Lecture 8: Image Segmentation Peng Chao Face++ Researcher - - PowerPoint PPT Presentation

▶
lecture 8 image segmentation
SMART_READER_LITE
LIVE PREVIEW

Lecture 8: Image Segmentation Peng Chao Face++ Researcher - - PowerPoint PPT Presentation

May 29, 2023 8 likes •1.05k views

Lecture 8: Image Segmentation Peng Chao Face++ Researcher pengchao@megvii.com Nov. 2017 Image Segmentation Semantic Segmentation Instance Segmentation Scene Parsing Human Parsing Stuff Segmentation New Track in COCO 2017

slide-1
SLIDE 1

Lecture 8: Image Segmentation

彭 超 Peng Chao Face++ Researcher pengchao@megvii.com

  • Nov. 2017
slide-2
SLIDE 2

Image Segmentation

slide-3
SLIDE 3

Semantic Segmentation

slide-4
SLIDE 4

Instance Segmentation

slide-5
SLIDE 5

Scene Parsing

slide-6
SLIDE 6

Human Parsing

slide-7
SLIDE 7

Stuff Segmentation

  • New Track in COCO 2017
  • Stuff: mountain, grass, wall, sky

……

  • Stuff covers about 66% of the

pixels in COCO

slide-8
SLIDE 8

UlrtraSound Segmentation

Figure credit: Ultrasound Nerve Segmentation on Kaggle

slide-9
SLIDE 9

Selfie Segmentation

slide-10
SLIDE 10

Evaluation

Normally, we use mean IOU to judge the results!

slide-11
SLIDE 11

Outline

  • Semantic Segmentation
  • Instance Segmentation
slide-12
SLIDE 12

Outline

  • Semantic Segmentation
  • Instance Segmentation
slide-13
SLIDE 13

Fully Convolutional Network

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015

slide-14
SLIDE 14

Fully Convolutional Network

Feature Map Downsampling Score Map Upsampling

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015

slide-15
SLIDE 15

Fully Convolutional Network

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015

slide-16
SLIDE 16

Fully Convolutional Network

  • First work using CNN to solve the semantic segmentation
  • Introducing skip-net framework
  • Large Improvement! (60 vs 30)

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015

slide-17
SLIDE 17

Learning Deconvolution Network for Semantic Segmentation

Noh, Hyeonwoo, Seunghoon Hong, and Bohyung Han. "Learning Deconvolution Network for Semantic Segmentation." ICCV 2015

slide-18
SLIDE 18

Learning Deconvolution Network for Semantic Segmentation

Noh, Hyeonwoo, Seunghoon Hong, and Bohyung Han. "Learning Deconvolution Network for Semantic Segmentation." ICCV 2015

slide-19
SLIDE 19

Learning Deconvolution Network for Semantic Segmentation

  • Introducing un-pool and de-convolution operations.
  • Introducing hourglass-like framework.

Noh, Hyeonwoo, Seunghoon Hong, and Bohyung Han. "Learning Deconvolution Network for Semantic Segmentation." ICCV 2015

slide-20
SLIDE 20

DeepLab

Liang-Chieh Chen*, George Papandreou*, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille (*equal contribution), arXiv preprint, 2016

slide-21
SLIDE 21

DeepLab

Liang-Chieh Chen*, George Papandreou*, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille (*equal contribution), arXiv preprint, 2016

slide-22
SLIDE 22

DeepLab

Liang-Chieh Chen*, George Papandreou*, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille (*equal contribution), arXiv preprint, 2016

slide-23
SLIDE 23

DeepLab

  • Introducing dilated-convolution
  • Combining traditional method (post processing): DenseCRF
slide-24
SLIDE 24

Conditional Random Field

Sutton, Charles A., and Andrew Mccallum. "An Introduction to Conditional Random Fields." arXiv: Machine Learning 4.4 (2012)

slide-25
SLIDE 25

y is the label, x is the image U: Unary relation; V: pairwise relation

Conditional Random Field

Sutton, Charles A., and Andrew Mccallum. "An Introduction to Conditional Random Fields." arXiv: Machine Learning 4.4 (2012)

slide-26
SLIDE 26

CRF Inference

❏ However, for loopy graph, the above problem is NP-hard. (The nodes relations are complex, making computing marginal probability harder) ❏ Approximated methods:

❏ MCMC (Gibbs Sampling) ❏ Loopy Belief propagation ❏ Mean Field

Sutton, Charles A., and Andrew Mccallum. "An Introduction to Conditional Random Fields." arXiv: Machine Learning 4.4 (2012)

slide-27
SLIDE 27

CRF Inference: Mean field

Sutton, Charles A., and Andrew Mccallum. "An Introduction to Conditional Random Fields." arXiv: Machine Learning 4.4 (2012)

slide-28
SLIDE 28

DenseCRF

Krahenbuhl, Philipp, and Vladlen Koltun. "Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials." NIPS 2011

slide-29
SLIDE 29

DenseCRF

Krahenbuhl, Philipp, and Vladlen Koltun. "Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials." NIPS 2011

slide-30
SLIDE 30

DenseCRF

  • Best Traditional Method!
  • Poor accuracy on segmentation! (poor feature)
slide-31
SLIDE 31

CRF AS RNN

Zheng, Shuai, et al. "Conditional Random Fields as Recurrent Neural Networks." ICCV 2015

slide-32
SLIDE 32

CRF AS RNN

Krahenbuhl, Philipp, and Vladlen Koltun. "Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials." NIPS 2011

slide-33
SLIDE 33

CRF AS RNN

  • Encoding the DenseCRF into the CNN framework!
  • Better results than DenseCRF in post-processing (Deeplab)
slide-34
SLIDE 34

Deeplab Attention

Attention to Scale: Scale-aware Semantic Image Segmentation CVPR 2016

slide-35
SLIDE 35

Deeplab Attention

Attention to Scale: Scale-aware Semantic Image Segmentation CVPR 2016

slide-36
SLIDE 36

Deeplab Attention

Attention to Scale: Scale-aware Semantic Image Segmentation CVPR 2016

slide-37
SLIDE 37

Deeplab Attention

  • Fusion framework for multi-scale training and inference!
  • Combining Attention model into Segmentation framework!
slide-38
SLIDE 38

PSPNet

Pyramid Secen Parsing Network

slide-39
SLIDE 39

PSPNet

Pyramid Secen Parsing Network

slide-40
SLIDE 40

PSPNet

Pyramid Secen Parsing Network

slide-41
SLIDE 41

PSPNet

  • Propose the Pyramid Pooling Module!
  • Hard to reproduce!
slide-42
SLIDE 42

Global Convolutional Network

TRF of two 3x3 conv is: 5 However the VRF maybe different！

Figure credit: Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the Inception Architecture for Computer Vision[J]. Computer Science, 2016. Zhou B, Khosla A, Lapedriza A, et al. Object Detectors Emerge in Deep Scene CNNs[J]. Computer Science, 2015.

slide-43
SLIDE 43

Global Convolutional Network

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, CVPR 2017

slide-44
SLIDE 44

Global Convolutional Network

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, CVPR 2017

slide-45
SLIDE 45

Global Convolutional Network

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, CVPR 2017

Baseline (FCN)

slide-46
SLIDE 46

Global Convolutional Network

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, CVPR 2017

Baseline (FCN) Gloabl Convolutional Network (GCN)

slide-47
SLIDE 47

Global Convolutional Network

Image GCN Baseline (FCN)

slide-48
SLIDE 48

Global Convolutional Network

Region Mis-Classifications are corrected!

Image GCN Baseline (FCN)

slide-49
SLIDE 49

Global Convolutional Network

Region Mis-Classifications are corrected! The Details are lost!

Image GCN Baseline (FCN)

slide-50
SLIDE 50

Boundary Refinement (BR)

slide-51
SLIDE 51

GCN GCN + BR Boundary Refinement (BR)

slide-52
SLIDE 52

GCN GCN + BR Boundary Refinement (BR) The Details are recoved!

slide-53
SLIDE 53

GCN GCN + BR Boundary Refinement (BR) Ground-Truth The Details are recoved!

slide-54
SLIDE 54
slide-55
SLIDE 55

Global Convolutional Network

  • Extend the FCN framework!
  • Partially Solve the Receptive Field Problem!
  • Two key components! (GCN and BRN)
slide-56
SLIDE 56

Deeplab V3

Rethinking Atrous Convolution for Semantic Image Segmentation, arxiv

slide-57
SLIDE 57

Deeplab V3

Rethinking Atrous Convolution for Semantic Image Segmentation, arxiv

slide-58
SLIDE 58

Deeplab V3

Rethinking Atrous Convolution for Semantic Image Segmentation, arxiv

slide-59
SLIDE 59

Deeplab V3

Rethinking Atrous Convolution for Semantic Image Segmentation, arxiv

slide-60
SLIDE 60

Deeplab V3

  • Currently State-Of-Art on PASCAL VOC 2012
  • Conclude the dilate-convolution technique on segmentation

Rethinking Atrous Convolution for Semantic Image Segmentation, arxiv

slide-61
SLIDE 61

Deformable Convolution

Deformable Convolutional Networks, arxiv

slide-62
SLIDE 62

Deformable Convolution

Deformable Convolutional Networks, arxiv

slide-63
SLIDE 63

Deformable Convolution

Deformable Convolutional Networks, arxiv

slide-64
SLIDE 64

Deformable Convolution

Deformable Convolutional Networks, arxiv

slide-65
SLIDE 65

Deformable Convolution

  • Solve the receptive field problem using learned offsets!
  • Also valid for detection!

Deformable Convolutional Networks, arxiv

slide-66
SLIDE 66

Re-Cap

  • Segmentation with CNN: FCN, Deeplab, GCN ...
  • Segmentation with CRF: DenseCRF, CRFAsRNN, ...
  • Different Convolutions: Dilated Conv, Global Conv, Deformable, ...
slide-67
SLIDE 67

Outline

  • Semantic Segmentation
  • Instance Segmentation
slide-68
SLIDE 68

Top-Down Pipeline

slide-69
SLIDE 69

Top-Down Pipeline

slide-70
SLIDE 70

Top-Down Pipeline

slide-71
SLIDE 71

FCIS

Fully Convolutional Instance-aware Semantic Segmentation, CVPR 2017

slide-72
SLIDE 72

FCIS

Fully Convolutional Instance-aware Semantic Segmentation, CVPR 2017

slide-73
SLIDE 73

FCIS

Fully Convolutional Instance-aware Semantic Segmentation, CVPR 2017

slide-74
SLIDE 74

FCIS

Fully Convolutional Instance-aware Semantic Segmentation, CVPR 2017

slide-75
SLIDE 75

FCIS

  • Bases on R-FCN framework
  • Propose the inside-outside scoring technique!
  • Winner of COCO InstanceSeg Challenge 2016!

Fully Convolutional Instance-aware Semantic Segmentation, CVPR 2017

slide-76
SLIDE 76

Mask RCNN

Mask-RCNN, ICCV 2017

slide-77
SLIDE 77

Mask RCNN

Mask-RCNN, ICCV 2017

slide-78
SLIDE 78

Mask RCNN

Mask-RCNN, ICCV 2017

slide-79
SLIDE 79

Mask RCNN

https://qiita.com/yu4u/items/5cbe9db166a5d72f9eb8

slide-80
SLIDE 80

Mask RCNN

https://qiita.com/yu4u/items/5cbe9db166a5d72f9eb8

slide-81
SLIDE 81

Mask RCNN

https://qiita.com/yu4u/items/5cbe9db166a5d72f9eb8

slide-82
SLIDE 82

Mask RCNN

Mask-RCNN, ICCV 2017

slide-83
SLIDE 83

Mask RCNN

  • State-Of-Art framework on InstanceSeg
  • Best paper of ICCV 2017!
  • Simple framework!

Mask-RCNN, ICCV 2017

slide-84
SLIDE 84

Bottom-up Pipeline

Semantic Instance Segmentation via Deep Metric Learning, arxiv

slide-85
SLIDE 85

Bottom-up Pipeline

Semantic Instance Segmentation via Deep Metric Learning, arxiv

slide-86
SLIDE 86

Bottom-up Pipeline

Semantic Instance Segmentation via Deep Metric Learning, arxiv

slide-87
SLIDE 87

Bottom-up Pipeline

Semantic Instance Segmentation via Deep Metric Learning, arxiv

slide-88
SLIDE 88

Bottom-up Pipeline

Semantic Instance Segmentation via Deep Metric Learning, arxiv

slide-89
SLIDE 89

Bottom-up Pipeline

  • Alternative framework to InstanceSeg
  • Tricky to implement
  • Incorporating the metric learning
slide-90
SLIDE 90

Top-Down

  • Instance ---> Segmentation

(for each instance)

  • MainStream
  • Start-Of-Art
  • Easy to implement
  • Difficulty: shrink the gap

between det and seg

  • Segmentation (for image)
  • --> instance
  • Alternative
  • Sub-Optimal
  • Tricky to implement
  • Difficulty: generate better

instance

Bottom-Up

slide-91
SLIDE 91

Re-Cap

  • Segmentation with CNN: FCN, Deeplab, GCN ...
  • Segmentation with CRF: DenseCRF, CRFAsRNN, ...
  • Different Convolutions: Dilated Conv, Global Conv, Deformable, ...
  • Top-Down pipeline for Instance Segmentation: FCIS, Mask-RCNN
  • Bottom-Up pipeline
slide-92
SLIDE 92

COCO & Places Challenge 2017

slide-93
SLIDE 93

COCO & Places Challenge 2017

slide-94
SLIDE 94

COCO & Places Challenge 2017

slide-95
SLIDE 95

COCO & Places Challenge 2017

slide-96
SLIDE 96

COCO & Places Challenge 2017

Track Rank Ensemble Single COCO BBox Detection 1st 52.8 50.5 Places InstanceSeg 1st 30.7 28.7 COCO Keypoint 1st 72.6 70.9 COCO InstanceSeg 2nd 46.4 45.0

slide-97
SLIDE 97

COCO Challenge 2017 BBOX

slide-98
SLIDE 98

COCO Challenge 2017 BBOX

Our Single Model is Here: 50.5.

slide-99
SLIDE 99

Places Challenge 2017 InstanceSeg

slide-100
SLIDE 100

COCO Challenge 2017 Keypoint

slide-101
SLIDE 101

COCO Challenge 2017 Keypoint

Google Research SenseTime

slide-102
SLIDE 102

COCO Challenge 2017 InstanceSeg

slide-103
SLIDE 103

Thanks