Segmentation and localization Kristen Grauman UT-Austin - - PDF document

▶
segmentation and localization
SMART_READER_LITE
LIVE PREVIEW

Segmentation and localization Kristen Grauman UT-Austin - - PDF document

Feb 19, 2023 248 likes •754 views

9/13/2016 Segmentation and localization Kristen Grauman UT-Austin Announcements Reminder: Assignment 1 due Friday Assignment 2 out today, due Sept 30 and followup Oct 3 Presenters: please send slides after class (naming

slide-1
SLIDE 1

9/13/2016 1

Segmentation and localization

Kristen Grauman UT-Austin

Announcements

  • Reminder: Assignment 1 due Friday
  • Assignment 2 out today, due Sept 30 and

followup Oct 3

  • Presenters: please send slides after class

(naming instructions on website)

slide-2
SLIDE 2

9/13/2016 2

Today: Mid-level cues

Tokens beyond pixels and filter responses but before object/scene categories

  • Edges, contours
  • Texture
  • Regions
  • Surfaces

Gradients -> edges

Primary edge detection steps:

  • 1. Smoothing: suppress noise
  • 2. Edge enhancement: filter for contrast
  • 3. Edge localization

Determine which local maxima from filter output are actually edges vs. noise

  • Threshold, Thin

Kristen Grauman

slide-3
SLIDE 3

9/13/2016 3

Original image Gradient magnitude image

slide-4
SLIDE 4

9/13/2016 4

Thresholding gradient with a lower threshold Thresholding gradient with a higher threshold

slide-5
SLIDE 5

9/13/2016 5

Canny edge detector

  • Filter image with derivative of Gaussian
  • Find magnitude and orientation of gradient
  • Non-maximum suppression:

– Thin wide “ridges” down to single pixel width

  • Linking and thresholding (hysteresis):

– Define two thresholds: low and high – Use the high threshold to start edge curves and the low threshold to continue them

  • MATLAB: edge(image, ‘canny’);
  • >>help edge

Source: D. Lowe, L. Fei-Fei

The Canny edge detector

  • riginal image (Lena)

Slide credit: Steve Seitz

slide-6
SLIDE 6

9/13/2016 6

The Canny edge detector

norm of the gradient

The Canny edge detector

thresholding

slide-7
SLIDE 7

9/13/2016 7

The Canny edge detector

thresholding How to turn these thick regions of the gradient into curves?

Non-maximum suppression

Check if pixel is local maximum along gradient direction, select single max across width of the edge

  • requires checking interpolated pixels p and r
slide-8
SLIDE 8

9/13/2016 8

The Canny edge detector

thinning (non-maximum suppression)

Problem: pixels along this edge didn’t survive the thresholding

Hysteresis thresholding

  • Use a high threshold to start edge curves,

and a low threshold to continue them.

Source: Steve Seitz

slide-9
SLIDE 9

9/13/2016 9

Hysteresis thresholding

  • riginal image

high threshold (strong edges) low threshold (weak edges) hysteresis threshold

Source: L. Fei-Fei

Hysteresis thresholding

  • riginal image

high threshold (strong edges) low threshold (weak edges) hysteresis threshold

Source: L. Fei-Fei

slide-10
SLIDE 10

9/13/2016 10

Recap: Canny edge detector

  • Filter image with derivative of Gaussian
  • Find magnitude and orientation of gradient
  • Non-maximum suppression:

– Thin wide “ridges” down to single pixel width

  • Linking and thresholding (hysteresis):

– Define two thresholds: low and high – Use the high threshold to start edge curves and the low threshold to continue them

  • MATLAB: edge(image, ‘canny’);
  • >>help edge

Source: D. Lowe, L. Fei-Fei

Background Texture Shadows

Low-level edges vs. perceived contours

Kristen Grauman

slide-11
SLIDE 11

9/13/2016 11

Texture

What defines a texture?

Includes: more regular patterns

slide-12
SLIDE 12

9/13/2016 12

Includes: more random patterns

Kristen Grauman

slide-13
SLIDE 13

9/13/2016 13

Kristen Grauman

http://animals.nationalgeographic.com/

Kristen Grauman

slide-14
SLIDE 14

9/13/2016 14

Texture representation

  • Textures are made up of repeated local

patterns, so:

– Find the patterns

  • Use filters that look like patterns (spots, bars, raw

patches…)

  • Consider magnitude of response

– Describe their statistics within each local window

  • Mean, standard deviation
  • Histogram
  • Histogram of “prototypical” feature occurrences

Kristen Grauman

Filter banks

  • What filters to put in the bank?

– Typically we want a combination of scales and orientations, different types of patterns.

Matlab code available for these examples: http://www.robots.ox.ac.uk/~vgg/research/texclass/filters.html

scales

  • rientations

“Edges” “Bars” “Spots”

slide-15
SLIDE 15

9/13/2016 15

Image from http://www.texasexplorer.com/austincap2.jpg

Kristen Grauman

Showing magnitude of responses

Kristen Grauman

slide-16
SLIDE 16

9/13/2016 16

Kristen Grauman Kristen Grauman

slide-17
SLIDE 17

9/13/2016 17

Kristen Grauman Kristen Grauman

slide-18
SLIDE 18

9/13/2016 18

Kristen Grauman Kristen Grauman

slide-19
SLIDE 19

9/13/2016 19

Kristen Grauman Kristen Grauman

slide-20
SLIDE 20

9/13/2016 20

Kristen Grauman Kristen Grauman

slide-21
SLIDE 21

9/13/2016 21

Kristen Grauman Kristen Grauman

slide-22
SLIDE 22

9/13/2016 22

Kristen Grauman Kristen Grauman

slide-23
SLIDE 23

9/13/2016 23

Kristen Grauman Kristen Grauman

slide-24
SLIDE 24

9/13/2016 24

Kristen Grauman Kristen Grauman

slide-25
SLIDE 25

9/13/2016 25

You try: Can you match the texture to the response?

Mean abs responses Filters A B C 1 2 3 Derek Hoiem

Representing texture by mean abs response

Mean abs responses Filters Derek Hoiem

slide-26
SLIDE 26

9/13/2016 26 [r1, r2, …, r38] We can form a feature vector from the list of responses at each pixel.

Kristen Grauman

  • Texton = cluster center of

filter responses over collection of images

  • Describe textures and

materials based on distribution of prototypical texture elements.

Textons

Leung & Malik 1999; Varma & Zisserman, 2002

slide-27
SLIDE 27

9/13/2016 27

Manik Varma http://www.robots.ox.ac.uk/~vgg/research/texclass/with.html

Materials as textures: example

Allows us to summarize an image according to its distribution of textons (prototypical texture patterns).

Varma & Zisserman, 2002

Materials as textures: example

Varma & Zisserman, 2002

slide-28
SLIDE 28

9/13/2016 28

http://www.airventure.org/2004/gallery/images/073104_satellite.jpg

Segmenting aerial imagery by textures

Kristen Grauman

Characterizing scene categories by texture

  • L. W. Renninger and
  • J. Malik. When is

scene identification just texture recognition? Vision Research 44 (2004) 2301–2311 Kristen Grauman

Scenes as textures: example

slide-29
SLIDE 29

9/13/2016 29

Texture: recap

  • Texture is a useful property that is often

indicative of materials, appearance cues

  • Texture representations attempt to summarize

repeating patterns of local structure

  • Filter banks useful to measure redundant

variety of structures in local neighborhood

Kristen Grauman

Mid-level cues

Tokens beyond pixels and filter responses but before object/scene categories

  • Edges, contours
  • Texture
  • Regions
  • Surfaces
slide-30
SLIDE 30

9/13/2016 30

Gestalt

  • Gestalt: whole or group

– Whole is greater than sum of its parts – Relationships among parts can yield new properties/features

  • Psychologists identified series of factors that

predispose set of elements to be grouped (by human visual system)

Similarity

http://chicagoist.com/attachments/chicagoist_alicia/GEESE.jpg, http://wwwdelivery.superstock.com/WI/223/1532/PreviewComp/SuperStock_1532R-0831.jpg

slide-31
SLIDE 31

9/13/2016 31

Symmetry

http://seedmagazine.com/news/2006/10/beauty_is_in_the_processingtim.php

Common fate

Image credit: Arthus-Bertrand (via F. Durand)

slide-32
SLIDE 32

9/13/2016 32

Proximity

http://www.capital.edu/Resources/Images/outside6_035.jpg

Some Gestalt factors

slide-33
SLIDE 33

9/13/2016 33

Illusory/subjective contours

In Vision, D. Marr, 1982

Interesting tendency to explain by occlusion

slide-34
SLIDE 34

9/13/2016 34

Continuity, explanation by occlusion

  • D. Forsyth
slide-35
SLIDE 35

9/13/2016 35

Figure-ground

slide-36
SLIDE 36

9/13/2016 36

In Vision, D. Marr, 1982; from J. L. Marroquin, “Human visual perception of structure”, 1976.

slide-37
SLIDE 37

9/13/2016 37

The goals of segmentation

Separate image into coherent “objects”

image human segmentation

Source: Lana Lazebnik

The goals of segmentation

Separate image into coherent “objects” Group together similar-looking pixels for efficiency of further processing

  • X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.

“superpixels”

Source: Lana Lazebnik

slide-38
SLIDE 38

9/13/2016 38

Segmentation as clustering

  • Families of clustering algorithms

– K-means – Mean shift – Graph cuts: normalized cuts, min-cut,… – Hierarchical agglomerative

Segmentation as clustering pixels

Depending on what we choose as the feature space, we can group pixels in different ways.

R=255 G=200 B=250 R=245 G=220 B=248 R=15 G=189 B=2 R=3 G=12 B=2 R G B

Grouping pixels based

  • n color similarity

Feature space: color value (3-d)

slide-39
SLIDE 39

9/13/2016 39

Segmentation as clustering pixels

  • Color, brightness, position alone are not

enough to distinguish all regions…

Segmentation with texture features

  • Find “textons” by clustering vectors of filter bank outputs
  • Describe texture in a window based on texton histogram

Malik, Belongie, Leung and Shi. IJCV 2001.

Texton map Image

Adapted from Lana Lazebnik

Texton index Texton index Count Count Count Texton index

slide-40
SLIDE 40

9/13/2016 40

Representing a texture gradient

Figure from Arbelaez et al PAMI 2011

g h

Contour Detection and Hierarchical Image Segmentation

Pablo Arbelaez, Michael Maire, Charless Fowlkes, Jitendra Malik Predict contours based on oriented gradients Map to closed regions with watershed Hierarchy of segments as output

slide-41
SLIDE 41

9/13/2016 41

[D. Martin et al. PAMI 2004]

Human-marked segment boundaries

Idea: learn from humans which combination of features is most indicative of a “good” contour [D. Martin et al. PAMI 2004]

What features are responsible for perceived edges?

Feature profiles (oriented energy, brightness, color, and texture gradients) along the patch’s horizontal diameter

slide-42
SLIDE 42

9/13/2016 42

[D. Martin et al. PAMI 2004]

What features are responsible for perceived edges?

Feature profiles (oriented energy, brightness, color, and texture gradients) along the patch’s horizontal diameter [D. Martin et al. PAMI 2004]

slide-43
SLIDE 43

9/13/2016 43

Computer Vision Group UC Berkeley

Contour Detection

Source: Jitendra Malik: http://www.cs.berkeley.edu/~malik/malik-talks-ptrs.html

Prewitt, Sobel, Roberts Canny Canny+opt thresholds Learned with combined features Human agreement

Ongoing topics in mid-level region representations

slide-44
SLIDE 44

9/13/2016 44

Multiple segmentations

  • Acknowledging difficulty of finding object boundaries in

single multi-way segmentation, now often employ multiple segmentations as “hypotheses”

  • Input to higher-level processes.

Hierarchy of segments

Fig from Maire et al. 2009

Greedy combinations

Fig from Hoiem et al. 2005

Varying parameters, grouping algorithms

Fig from Russell et al. 2006

Segments as primitives for discovery

  • B. Russell et al., “Using Multiple Segmentations to Discover Objects and

their Extent in Image Collections,” CVPR 2006

Multiple segmentations

slide-45
SLIDE 45

9/13/2016 45

Segments as object parts?

Gu et al. Recognition Using Regions, CVPR 2009

Top-down segmentation

Slide credit: Lana Lazebnik

  • E. Borenstein and S. Ullman, “Class-specific, top-down segmentation,” ECCV 2002
  • A. Levin and Y. Weiss, “Learning to Combine Bottom-Up and Top-Down Segmentation,”

ECCV 2006.

slide-46
SLIDE 46

9/13/2016 46

Top-down segmentation

  • E. Borenstein and S. Ullman, “Class-specific, top-down segmentation,” ECCV 2002
  • A. Levin and Y. Weiss, “Learning to Combine Bottom-Up and Top-Down Segmentation,”

ECCV 2006. Normalized cuts Top-down segmentation

Slide credit: Lana Lazebnik

Motion segmentation

Image Segmentation Motion Segmentation Input sequence Image Segmentation Motion Segmentation Input sequence

A.Barbu, S.C. Zhu. Generalizing Swendsen-Wang to sampling arbitrary posterior probabilities, IEEE Trans. PAMI, August 2005.

slide-47
SLIDE 47

9/13/2016 47

Regions to surfaces

Learn to categorize regions into geometric classes Combining multiple segmentations

Geometric Context from a Single Image. Derek Hoiem, Alexei Efros, Martial Hebert. ICCV 2005

  • Ladicky, Zeisl, Pollefeys. Discriminatively Trained Dense Surface Normal
  • Estimation. ECCV 2014

Regions to surfaces

Predicting surface normals

slide-48
SLIDE 48

9/13/2016 48

Category-independent ranking

How “object-like” is each candidate region?

Constrained Parametric Min-Cuts for Automatic Object Segmentation. Carreira and Sminchisescu. CVPR 2010

Also see Ferrari et al CVPR 2010, Endres et al ECCV 2010 [Jain & Grauman, Supervoxel-Consistent Foreground Propagation in Video, ECCV 2014]

Video object segmentation

Kristen Grauman, UTCS

slide-49
SLIDE 49

9/13/2016 49

Video object segmentation

Fanyi Xiao and Yong Jae Lee Track and Segment: An Iterative Unsupervised Approach for Video Object Proposals In CVPR 2016