Context For Semantic Segmentation Gang Yu Collaborators - - PowerPoint PPT Presentation

▶
context for semantic segmentation
SMART_READER_LITE
LIVE PREVIEW

Context For Semantic Segmentation Gang Yu Collaborators - - PowerPoint PPT Presentation

Jul 25, 2023 329 likes •777 views

Context For Semantic Segmentation Gang Yu Collaborators Changqian Yu Jingbo Wang Chao Peng Xiangyu Zhang Changxin Gao Nong Sang Gang Yu Jian Sun Outline Revisit Semantic Segmentation Context for Semantic

slide-1
SLIDE 1

Gang Yu 旷 视 研 究 院

Context For Semantic Segmentation

slide-2
SLIDE 2

Chao Peng Jingbo Wang Changqian Yu Changxin Gao Xiangyu Zhang Gang Yu Jian Sun

Collaborators

Nong Sang

slide-3
SLIDE 3

Outline

  • Revisit Semantic Segmentation
  • Context for Semantic Segmentation
  • Backbone
  • Head
  • Loss
  • Conclusion
slide-4
SLIDE 4

Outline

  • Revisit Semantic Segmentation
  • Context for Semantic Segmentation
  • Backbone
  • Head
  • Loss
  • Conclusion
slide-5
SLIDE 5

What is Semantic Segmentation?

  • Classification + Localization
  • Visual Recognition
  • Classification
  • Semantic Segmentation
  • Instance Segmentation
  • Panoptic Segmentation
  • Detection
  • Keypoint Detection
slide-6
SLIDE 6

Pipeline

Backbone

Head

LOSS

VGG16 ResNet ResNext … Softmax L2 … U-Shape 4/8-Sampling + Dilation …

slide-7
SLIDE 7

Challenges in Semantic Segmentation?

  • Speed
  • Performance
  • Per-pixel Accuracy
  • Boundary
slide-8
SLIDE 8

What is Context?

  • According to Dictionary:
  • the parts of a discourse that surround a word or passage and

can throw light on its meaning

Sports ball Grass Play Fields Person

slide-9
SLIDE 9

Outline

  • Revisit Semantic Segmentation
  • Context for Semantic Segmentation
  • Backbone
  • Head
  • Loss
  • Conclusion
slide-10
SLIDE 10

Context in Backbone

  • Motivation
  • Traditional Backbone is designed for Classification
  • Large Receptive field by compromising spatial resolution
  • Segmentation requires both Classification & Localization
  • Maintain both Receptive Field (context) & Spatial resolution
  • Computational cost?
slide-11
SLIDE 11

Context in Backbone - BiSeNet

BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, ECCV, 2018

  • BiSeNet: Bilateral Segmentation Network
slide-12
SLIDE 12

Context in Backbone - BiSeNet

BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, ECCV, 2018

  • Pipeline
slide-13
SLIDE 13

Context in Backbone - BiSeNet

BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, ECCV, 2018

  • Results
slide-14
SLIDE 14

Context in Backbone - BiSeNet

BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, ECCV, 2018

  • Ablation Results
slide-15
SLIDE 15

Context in Backbone - BiSeNet

BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, ECCV, 2018

  • Speed
slide-16
SLIDE 16

Context in Backbone - BiSeNet

BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, ECCV, 2018

  • Summary
  • Two path in backbone: Spatial path + Context path
  • Context is implicitly encoded in receptive field
  • Efficient speed
  • Code: https://github.com/ycszen/TorchSeg
  • Context:
  • A branch encodes semantic meaning with large receptive field?
  • Related work:
  • ICNet for Real-Time Semantic Segmentation on High-Resolution Images, Hengshuang Zhao,

Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, Jiaya Jia, ECCV2018

  • Stacked Hourglass Networks for Human Pose Estimation, Alejandro Newell, Kaiyu Yang, Jia

Deng, ECCV2016

slide-17
SLIDE 17

Context in Head

  • Motivation
  • Large Receptive field without compromising boundary results
  • Why working on Head?
  • Efficient speed
  • Obvious gain on increasing the receptive
  • Simple to implement
slide-18
SLIDE 18

Context in Head – Large Kernel

  • Receptive Field vs Valid Receptive Field

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-19
SLIDE 19

Context in Head – Large Kernel

  • Large Kernel Matters

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-20
SLIDE 20

Context in Head – Large Kernel

  • Large Kernel Matters
  • Why Boundary Refinement?
  • Large receptive field will blur the object boundary

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-21
SLIDE 21

Context in Head – Large Kernel

  • Large Kernel Matters
  • Ablation: Why Boundary Refinement?
  • Large receptive field will blur the object boundary

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-22
SLIDE 22

Context in Head – Large Kernel

  • Large Kernel Matters
  • Ablation: Different kernel size?

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-23
SLIDE 23

Context in Head – Large Kernel

  • Large Kernel Matters
  • Ablation: Are more parameters helpful?

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-24
SLIDE 24

Context in Head – Large Kernel

  • Large Kernel Matters
  • Ablation: GCN vs. Stack of small convolutions

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-25
SLIDE 25

Context in Head – Large Kernel

  • Large Kernel Matters
  • Ablation: GCN in Backbone

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-26
SLIDE 26

Context in Head – Large Kernel

  • Large Kernel Matters: illustrative examples

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-27
SLIDE 27

Context in Head – Large Kernel

  • Summary
  • Global Convolution network to increase the receptive field
  • Large separable convolution is an efficient implementation
  • Context
  • Large receptive field?
  • Related work
  • PSPNet: Pyramid Scene Parsing Network, Hengshuang Zhao, Jianping Shi, Xiaojuan Qi,

Xiaogang Wang, Jiaya Jia, CVPR2017

  • DeeplabV3: Rethinking Atrous Convolution for Semantic Image Segmentation, Liang-Chieh

Chen, George Papandreou, Florian Schroff, Hartwig Adam

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

slide-28
SLIDE 28

Context in Head – DFN

  • Motivation:
  • Large kernel (GCN) is computationally intensive
  • Global pooling is efficient to compute and can obtain the

global context

  • Large receptive field does not equal to good context
  • Attention strategy to adaptively aggreate the features

Learning a Discriminative Feature Network for Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, CVPR, 2018

slide-29
SLIDE 29

Context in Head – DFN

  • DFN: Pipeline

Learning a Discriminative Feature Network for Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, CVPR, 2018

slide-30
SLIDE 30

Context in Head – DFN

  • DFN: Ablation

Learning a Discriminative Feature Network for Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, CVPR, 2018

slide-31
SLIDE 31

Context in Head – DFN

  • DFN: Results

Learning a Discriminative Feature Network for Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, CVPR, 2018

slide-32
SLIDE 32

Context in Head – DFN

  • Summary
  • Global pooling is efficient and effective to capture the long-range

context

  • Attention for adaptive adjusting feature weights
  • Code: https://github.com/ycszen/TorchSeg/
  • Context
  • Receptive field & feature aggregation?
  • Related work
  • Non-local Neural Networks, Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He, CVPR2018
  • CCNet: Criss-Cross Attention for Semantic Segmentation, Zilong Huang, Xinggang Wang, Lichao

Huang, Chang Huang, Yunchao Wei, Wenyu Liu

  • PSANet: Point-wise Spatial Attention Network for Scene Parsing, Hengshuang Zhao*, Yi Zhang*, Shu

Liu, Jianping Shi, Chen Change Loy, Dahua Lin, Jiaya Jia, ECCV2018

  • OCNet: Object Context Network for Scene Parsing, Yuhui Yuan, Jingdong Wang
  • ParseNet: Looking Wider to See Better, Wei Liu, Andrew Rabinovich, Alexander C. Berg

Learning a Discriminative Feature Network for Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, CVPR, 2018

slide-33
SLIDE 33

Context in Loss

  • Motivation
  • “Thing” may be important for stuff prediction

COCO2018 Panoptic Segmentation Challenge, http://presentations.cocodataset.org/ECCV18/COCO18-Panoptic-Megvii.pdf

Sports ball Grass Play Fields Person

slide-34
SLIDE 34

Context in Loss

  • Motivation
  • “Thing” may be important for stuff prediction

COCO2018 Panoptic Segmentation Challenge, http://presentations.cocodataset.org/ECCV18/COCO18-Panoptic-Megvii.pdf

slide-35
SLIDE 35

Encoder Train/Inference Train Supervision Inference Merge

Res-Block

Multi Types Context

Objects Semantic Stuff Stuff

Context in Loss

  • Pipeline

COCO2018 Panoptic Segmentation Challenge, http://presentations.cocodataset.org/ECCV18/COCO18-Panoptic-Megvii.pdf

slide-36
SLIDE 36

Context in Loss

  • COCO2018 Panoptic Segmentation Challenge

49.3 49.6 54.1 54.5 50.8 Res50 +Encoder +Extra Res Blocks +Multi Context +Huge Backbone +Multi-Scale Flip Test Results of Stuff Regions on COCO2018 Panoptic Segmentation Validation Dataset Metric：Mean IoU% Finally, we assembled three models and achieve 55.9% mIoU on this dataset.

COCO2018 Panoptic Segmentation Challenge, http://presentations.cocodataset.org/ECCV18/COCO18-Panoptic-Megvii.pdf

slide-37
SLIDE 37

Context in Loss

  • COCO2018 Panoptic Segmentation Challenge

COCO2018 Panoptic Segmentation Challenge, http://presentations.cocodataset.org/ECCV18/COCO18-Panoptic-Megvii.pdf

slide-38
SLIDE 38

Context in Loss

  • Summary
  • “Thing” and “stuff” are complementary
  • Loss is a good approach to encode the context
  • Better feature representation
  • Context
  • A loss to encode the semantic meaning?
  • Related work
  • Context Encoding for Semantic Segmentation, Hang Zhang, Kristin Dana, Jianping Shi,

Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, Amit Agrawal, CVPR2018

COCO2018 Panoptic Segmentation Challenge, http://presentations.cocodataset.org/ECCV18/COCO18-Panoptic-Megvii.pdf

slide-39
SLIDE 39

Outline

  • Revisit Semantic Segmentation
  • Context for Semantic Segmentation
  • Backbone
  • Head
  • Loss
  • Conclusion
slide-40
SLIDE 40

Conclusion

  • Context in different parts
  • Backbone, Head, Loss
  • What is Context?
  • Large receptive field?
  • A semantic branch?
  • Spatial/feature aggregation?
  • Future work
  • Explicitly show what is a context
  • Panoptic seg: Stuff vs Thing
slide-41
SLIDE 41

Reference

  • Pyramid Scene Parsing Network, Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia,

CVPR2017

  • ICNet for Real-Time Semantic Segmentation on High-Resolution Images, Hengshuang Zhao, Xiaojuan Qi,

Xiaoyong Shen, Jianping Shi, Jiaya Jia, ECCV2018

  • Context Encoding for Semantic Segmentation, Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang,

Xiaogang Wang, Ambrish Tyagi, Amit Agrawal, CVPR2018

  • Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, Liang-Chieh Chen,

Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam, ECCV2018

  • Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network, Chao Peng,

Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun, CVPR, 2017

  • Learning a Discriminative Feature Network for Semantic Segmentation, Changqian Yu, Jingbo Wang, Chao

Peng, Changxin Gao, Gang Yu, Nong Sang, CVPR, 2018

  • BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation, Changqian Yu, Jingbo

Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang, ECCV, 2018

slide-42
SLIDE 42

Q&A

  • Megvii Detection 知乎专栏 
  • Webpage: http://www.skicyyu.org/
  • Email: yugang@megvii.com
slide-43
SLIDE 43