free 18-May-17 Towards Weakly Supervised Image Understanding 1/50 - - PowerPoint PPT Presentation

▶
free
SMART_READER_LITE
LIVE PREVIEW

free 18-May-17 Towards Weakly Supervised Image Understanding 1/50 - - PowerPoint PPT Presentation

Sep 23, 2023 284 likes •809 views

free 18-May-17 Towards Weakly Supervised Image Understanding 1/50 Towards Weakly Supervised Image Understanding (WSIU) http://mmcheng.net/ 18-May-17 Towards Weakly

slide-1
SLIDE 1 Towards Weakly Supervised Image Understanding 18-May-17 1/50

free

slide-2
SLIDE 2 Towards Weakly Supervised Image Understanding 18-May-17 2/50

Towards Weakly Supervised Image Understanding (WSIU)

报告人：程明明 南开大学、计算机与控制工程学院 http://mmcheng.net/

slide-3
SLIDE 3 Towards Weakly Supervised Image Understanding 18-May-17 3/50

Understanding Visual Information

Image by kirkh.deviantart.com

slide-4
SLIDE 4 Towards Weakly Supervised Image Understanding 18-May-17 4/50

Dataset Annotation

slide-5
SLIDE 5 Towards Weakly Supervised Image Understanding 18-May-17 5/50

Dataset Annotation

CVML 2012, Antonio Torralba

  • PASCAL 11:
  • 10? workers
  • 27.374 bounding boxes
  • ImageNet:
  • 25.000 workers
  • 11.231.732 images labeled

with one word

  • My mother:
  • 213.841 segmented objects
  • Job offer: I am looking for

more parents

slide-6
SLIDE 6 Towards Weakly Supervised Image Understanding 18-May-17 6/50

Dataset Annotation

slide-7
SLIDE 7 Towards Weakly Supervised Image Understanding 18-May-17 7/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-8
SLIDE 8 Towards Weakly Supervised Image Understanding 18-May-17 8/50

Outline of the survey

  • Low level vision
  • Attention: CVPR 11, CVPR 14, TPAMI 15, IEEE TIP 15, IJCV 17,

CVPR 17

  • Boundary: CVPR 17
  • Segmentation: TVC 14, CGF 15, ECCV 15, TPAMI 15
  • Light weighted semantic parsing
  • Semantic segmentation: TPAMI 17, CVPR (oral) 17
  • Interaction: TOG 15, TOG 10, TOG 12
  • Graphics/vision applications
  • Editing: TOG 14
  • Synthesis: TOG 09
  • Web images: CVPR 12, TOG 11, CVPR 12, CVPR 13, TOG 12
slide-9
SLIDE 9 Towards Weakly Supervised Image Understanding 18-May-17 9/50

Visual attention: motivation

slide-10
SLIDE 10 Towards Weakly Supervised Image Understanding 18-May-17 10/50

Visual attention

Deeply supervised salient object detection with short connections, IEE EEE CVP CVPR 2017 2017. Salient Object Detection: A Discriminative Regional Feature Integration Approach, IJCV JCV 2017 2017. Salient Object Detection: A Benchmark, IEE EEE TIP TIP 2015 2015. Global Contrast based Salient Region Detection, IEE EEE TP TPAMI 2015 2015. BING: Binarized Normed Gradients for Objectness Estimation at 300fps, IEE EEE CVP CVPR 2014 2014.

fixation prediction Objectness proposals Salient object detection

slide-11
SLIDE 11 Towards Weakly Supervised Image Understanding 18-May-17 11/50 Global Contrast based Salient Region Detection, IEEE TPAMI, 2014, MM Cheng, et. al. (2nd nd most cited pap paper in in CVP CVPR 2011 2011)
slide-12
SLIDE 12 Towards Weakly Supervised Image Understanding 18-May-17 12/50

Core idea: region contrast (RC)

Region size Image Segmentation 𝜏𝑡

2 → ∞

𝜏𝑡

2 → 0.4

Spatial weighting

𝑇 𝑠

𝑙 = 𝑠𝑙≠𝑠𝑗 exp − 𝐸𝑡 𝑠𝑙,𝑠𝑗 𝜏𝑡

2

𝜕 𝑠

𝑗 𝐸𝑠(𝑠 𝑙, 𝑠 𝑗) Region contrast by sparse histogram comparison.

slide-13
SLIDE 13 Towards Weakly Supervised Image Understanding 18-May-17 13/50

Experimental results

  • Dataset: MSRA1000 [Achanta09]
  • Precision vs. recall
slide-14
SLIDE 14 Towards Weakly Supervised Image Understanding 18-May-17 14/50

Supervised feature integration

Salient Object Detection: A Discriminative Regional Feature Integration Approach, IJCV JCV 2017 2017.
slide-15
SLIDE 15 Towards Weakly Supervised Image Understanding 18-May-17 15/50

Benchmarking 40+ methods

Salient Object Detection: A Benchmark, IEE EEE TIP TIP 2015 2015.
slide-16
SLIDE 16 Towards Weakly Supervised Image Understanding 18-May-17 16/50

Going with deep models

Deeply supervised salient object detection with short connections, IEE EEE CVP CVPR 2017 2017.
slide-17
SLIDE 17 Towards Weakly Supervised Image Understanding 18-May-17 17/50

Bridging between multi-levels

slide-18
SLIDE 18 Towards Weakly Supervised Image Understanding 18-May-17 18/50

Messages from numbers

slide-19
SLIDE 19 Towards Weakly Supervised Image Understanding 18-May-17 19/50

Methodology: observation

  • Objects are stand-alone things with well defined closed

boundaries and centers.

  • Little variations could present in such abstracted view.
Finding pictures of objects in large collections of images. Springer Berlin Heidelberg, 1996, Forsyth et. al. Using stuff to find things. ECCV 2008, Heitz et. al. Measuring the objectness of image window, IEEE TPAMI 2012, Alexe et. al.
slide-20
SLIDE 20 Towards Weakly Supervised Image Understanding 18-May-17 20/50

Experimental results

  • Proposal quality on PASCAL VOC 2007
slide-21
SLIDE 21 Towards Weakly Supervised Image Understanding 18-May-17 21/50

Experimental results

  • Computational time
  • A laptop with an Intel i7-3940XM CPU
  • 20 seconds for training on the PASCAL 2007 training set!!
  • Testing time 300fps on VOC 2007 images

Method [1] OBN [2] CSVM [3] SEL [4] Our BING Time (seconds) 89.2 3.14 1.32 11.2 0.003

Category-Independent Object Proposals With Diverse Ranking, PAMI 2014, Endres et. al. Measuring the objectness of image windows. PAMI 2012, Alexe, et. al. Proposal Generation for Object Detection using Cascaded Ranking SVMs. CVPR 2011, Zhang et. al. Selective Search for Object Recognition, IJCV 2013, Uijlings et. al.
slide-22
SLIDE 22 Towards Weakly Supervised Image Understanding 18-May-17 22/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-23
SLIDE 23 Towards Weakly Supervised Image Understanding 18-May-17 23/50

Boundary

Richer Convolutional Features for Edge Detection, IEE EEE CVP CVPR 2017 2017.
slide-24
SLIDE 24 Towards Weakly Supervised Image Understanding 18-May-17 24/50
slide-25
SLIDE 25 Towards Weakly Supervised Image Understanding 18-May-17 25/50

Sate of the arts

slide-26
SLIDE 26 Towards Weakly Supervised Image Understanding 18-May-17 26/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-27
SLIDE 27 Towards Weakly Supervised Image Understanding 18-May-17 27/50

SaliencyCut

  • Iterative refine: iteratively run GrabCut to refine segmentation
  • Adaptive fitting: adaptively fit with newly segmented salient region

Enables automatic initialization provided by salient object detection.

Global Contrast based Salient Region Detection, IEE EEE TP TPAMI 2015 2015.
slide-28
SLIDE 28 Towards Weakly Supervised Image Understanding 18-May-17 28/50

Salient shape

  • Is salient object detection for ‘simple’ images useful?
SalientShape: Group Saliency in Image Collections, TV TVC 2016 2016.
slide-29
SLIDE 29 Towards Weakly Supervised Image Understanding 18-May-17 29/50

Segmentation

DenseCut: Densely Connected CRFs for Realtime GrabCut, CGF GF 2015 2015.
slide-30
SLIDE 30 Towards Weakly Supervised Image Understanding 18-May-17 30/50

Segmentation

HFS: Hierarchical Feature Selection for Efficient Image Segmentation, ECC CCV 2015 2015.
slide-31
SLIDE 31 Towards Weakly Supervised Image Understanding 18-May-17 31/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-32
SLIDE 32 Towards Weakly Supervised Image Understanding 18-May-17 32/50

STC

STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation, arXiv 2015, Wei et al.

10% improvement

  • ver state of the art!
slide-33
SLIDE 33 Towards Weakly Supervised Image Understanding 18-May-17 33/50

Interaction

Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach, IEE EEE CVP CVPR (or (oral) al) 2017 2017.
slide-34
SLIDE 34 Towards Weakly Supervised Image Understanding 18-May-17 34/50

Results

slide-35
SLIDE 35 Towards Weakly Supervised Image Understanding 18-May-17 35/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-36
SLIDE 36 Towards Weakly Supervised Image Understanding 18-May-17 36/50

Motivation

Pixels/patches Objects Scene Layering …

slide-37
SLIDE 37 Towards Weakly Supervised Image Understanding 18-May-17 37/50

Motivation

  • Rough depth ordering is possible for a single image

with repeated elements

RepFinder: Finding Approximately Repeated Scene Elements for Image Editing, ACM CM TOG OG 2010 2010.
slide-38
SLIDE 38 Towards Weakly Supervised Image Understanding 18-May-17 38/50

Image rearrangement

Source image

slide-39
SLIDE 39 Towards Weakly Supervised Image Understanding 18-May-17 39/50
slide-40
SLIDE 40 Towards Weakly Supervised Image Understanding 18-May-17 40/50
slide-41
SLIDE 41 Towards Weakly Supervised Image Understanding 18-May-17 41/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-42
SLIDE 42 Towards Weakly Supervised Image Understanding 18-May-17 42/50 ImageSpirit: Verbal Guided Image Parsing, ACM TOG, 2014
slide-43
SLIDE 43 Towards Weakly Supervised Image Understanding 18-May-17 43/50

Motivations

slide-44
SLIDE 44 Towards Weakly Supervised Image Understanding 18-May-17 44/50

Verbal guided image parsing

Make the wood cabinet in bottom-middle lower nouns Adjective Verb/Adverb Multi label CRF

Object Attributes Commands

slide-45
SLIDE 45 Towards Weakly Supervised Image Understanding 18-May-17 45/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-46
SLIDE 46 Towards Weakly Supervised Image Understanding 18-May-17 46/50

Sketch2Photo

Sketch2photo: internet image montage, ACM TOG 2009.

slide-47
SLIDE 47 Towards Weakly Supervised Image Understanding 18-May-17 47/50

Towards WSIU

Low level vision

Attention Segmentation Boundary

Light weighted semantic parsing

Semantic segmentation Interaction

Graphics/vision applications

Editing Web images Synthesis

slide-48
SLIDE 48 Towards Weakly Supervised Image Understanding 18-May-17 48/50

Dealing with web images

Columbia, Shihfu Chang CVPR 12 MIT, Rubinstein CVPR 13 NUS Shuicheng Yan PAMI 17 UCSD, Zhuowen Tu, CVPR 12 北理工 黄华 ACM TOG 11 NUS, Ping Tan ACM TOG 11

slide-49
SLIDE 49 Towards Weakly Supervised Image Understanding 18-May-17 49/50

Media Computing Lab @ Nankai

  • Visiting Professors
  • Collaborators
slide-50
SLIDE 50 Towards Weakly Supervised Image Understanding 18-May-17 50/50

Students (2016 - )

  • Qibin Hou
  • Deeply supervised salient object detection …, Q Hou, MM

Cheng, X Hu, Z Tu, A Borji, IEEE CVPR, 2017.

  • Intelligent Visual Media Processing: When Graphics Meets

Vision, MM Cheng, Q Hou, SH Zhang, PL Rosin. JCST, 2017.

  • Yun Liu
  • Richer Convolutional Features for Edge Detection, Y Liu, MM

Cheng, X Hu, K Wang, X Bai, IEEE CVPR, 2017.

  • HFS: Hierarchical Feature Selection for ... MM Cheng, Y Liu, Q

Hou, J Bian, P Torr, SM Hu, Z Tu. ECCV, 2016.

  • Jia-Wang Bian
  • GMS: Grid-based Motion … Feature correspondence, JW Bian,

W Lin, … , MM Cheng IEEE CVPR, 2017.

slide-51
SLIDE 51 Towards Weakly Supervised Image Understanding 18-May-17 51/50

Thanks!