Lecture 21: Grouping 1 Announcements Final project guidelines and - - PowerPoint PPT Presentation

▶
lecture 21 grouping
SMART_READER_LITE
LIVE PREVIEW

Lecture 21: Grouping 1 Announcements Final project guidelines and - - PowerPoint PPT Presentation

Nov 04, 2023 472 likes •1.38k views

Lecture 21: Grouping 1 Announcements Final project guidelines and sign-up sheet out on Piazza Writeup (70%) Due 4/26 midnight Grading rubric provided Presentation (20%) Sign-up sheet for 4/27 - 4/28. If no times

slide-1
SLIDE 1

Lecture 21: Grouping

1

slide-2
SLIDE 2

Announcements

  • Final project guidelines and sign-up sheet out on Piazza


Writeup (70%)

  • Due 4/26 midnight
  • Grading rubric provided


Presentation (20%)

  • Sign-up sheet for 4/27 - 4/28.
  • If no times work, send private Piazza post ASAP

.

  • 5 mins + 2 mins of questions.
  • Recommend splitting time between teammates
slide-3
SLIDE 3

Announcements

  • Final project guidelines and sign-up sheet out


Writeup (70%)

  • Should have intro, related work, method, experiments,

conclusion.

  • At most 6 pages in CVPR format
  • Evaluated on:
  • Background: how well do you explain the related

work?

  • Completeness: results quality, what worked or didn’t

work, thoroughness of evaluation

  • Format and clarity
slide-4
SLIDE 4

Today

  • Image segmentation
  • Edge-aware image processing

4

slide-5
SLIDE 5

Recall: semantic segmentation problem

f

<latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit><latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit><latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit><latexit sha1_base64="b4HLEbhr7TEtaehBb4ygFYyuiV8=">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm/GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipGQ7KFbfqLkDWiZeTCuRoDMpf/WHM0gilYJq3fPcxPgZVYzgbNSP9WYUDahI+xZKmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUWZsNiUbgrf68jpX1U9t+o1ryv12zyOIpzBOVyCBzWowz0oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8AyvGM6g=</latexit>

(Colors represent categories)

Source: Torralba, Freeman, Isola

5

slide-6
SLIDE 6

Bottom-up segmentation

  • Group together similar-looking pixels
  • “Bottom-up” process
  • Unsupervised

6

Source: A. Oliva

slide-7
SLIDE 7

A “simple” segmentation problem

7 Source: Torralba and Freeman

slide-8
SLIDE 8

Segmentation is a global process

What are the occluded numbers?

Source: A. Oliva 8

slide-9
SLIDE 9

Occlusion is an important cue in grouping.

Segmentation is a global process

What are the occluded numbers?

Source: A. Oliva 9

slide-10
SLIDE 10

Groupings by Invisible Completions

* Images from Steve Lehar’s Gestalt papers

Source: A. Oliva 10

slide-11
SLIDE 11

… but not too global

Source: A. Oliva 11

slide-12
SLIDE 12

Photo credit: R. C. James

12

slide-13
SLIDE 13

Perceptual organization

“…the processes by which the bits and pieces of visual information that are available in the retinal image are structured into the larger units of perceived objects and their interrelations” Stephen E. Palmer, Vision Science, 1999

Source: A. Oliva 13

slide-14
SLIDE 14

Gestalt principles

There are hundreds of different grouping laws

14

slide-15
SLIDE 15

Source: A. Oliva 15

slide-16
SLIDE 16

Familiar configuration

Source: A. Oliva 16

slide-17
SLIDE 17

Clustering

Source: K. Grauman 17

slide-18
SLIDE 18

A simple segmentation algorithm

  • Each pixel is described by a vector

z = [r, g, b] or [Y u v], …

  • Run a clustering algorithm (e.g. k-means)

using some distance between pixels: D(pixel i, pixel j) = || zi – zj ||2

Source: A. Oliva 18

slide-19
SLIDE 19

19

Method 1: K-Means

Example from: https://en.wikipedia.org/wiki/K-means_clustering

Idea: find K centroids that cover the points

slide-20
SLIDE 20

20

K-Means

Example from: https://en.wikipedia.org/wiki/K-means_clustering

arg min

S k

X

i=1

X

x∈Si

kx µik2

<latexit sha1_base64="stxZofKDEFZTyhgykY20wKv2T5Q=">ACknicbVHLbhMxFPUMrxIeDY8dG4sIiQVEMwUEQgIK3bBgUVTSVorTyOPxJFb8GNl3ECMzH8TvsONvuJMGVFquZPn4nHPte6+LWqsAWfYrS9dvnL12tb1wY2bt25vD+/cPQyu8UJOhNPOHxc8SK2snIACLY9rL7kptDwqVnu9fvRV+qCc/QJtLWeGL6yqlOCA1Hz4I7JShVrzNkCrJY2sSX6JSA0HJZFReNBR7vIXC09B+ctN+jfsGeGVR6lhoDJ1H9SbvTuLqz/Fv+reOMmXpATrQq2UF7PtZ8WlkhdNlaA1u/bMN3tmbmVeLJZpP4g4Sw1E2ztZBL4J8A0ZkE/vz4U9WOtEYaUFoHsI0z2qYeGghJbdgDVB1lys+EJOEfZthVlcj7Sj5ApaeU8Lgt0zZ7NiNyEvmB09p2E81pP/k+bNlC9mkVl6wakFacPVY2m4Gj/P7RUXgrQLQIuvMJaqVhyzwXgrwxwCPn5li+Cw51x/mycfX4+2v2wGcWeUAeksckJy/JLvlI9smEiGQ7eZG8Td6l9PX6ft079SaJpuce+SfSD/9Bn+YzKo=</latexit>
  • Points are assigned to nearest centroid
  • Find centroids µi that minimize distance to points assigned to it

Minimize:

slide-21
SLIDE 21

21

K-Means

Example from: https://en.wikipedia.org/wiki/K-means_clustering

  • 1. Initialize centroids randomly
slide-22
SLIDE 22

22

K-Means

Example adapted from: https://en.wikipedia.org/wiki/K-means_clustering

  • 2. Assign points to nearest cluster centroid
slide-23
SLIDE 23

23

K-Means

Example adapted from: https://en.wikipedia.org/wiki/K-means_clustering

  • 3. Update centroids
slide-24
SLIDE 24

24

K-Means

Example adapted from: https://en.wikipedia.org/wiki/K-means_clustering

Repeat until change is very small

slide-25
SLIDE 25

K-Means Clustering

  • Given k, the k-means algorithm consists of four steps:

25

– Select initial centroids at random. – Assign each point to the cluster with the nearest centroid. – Compute each centroid as the mean of the points assigned to it. – Repeat previous 2 steps until no change.

Source: A. Oliva

slide-26
SLIDE 26

Image Intensity-based clusters Color-based clusters

  • K-means (k=5) clustering based on intensity

(middle) or color (right) is essentially vector quantization of the image attributes – Clusters don’t have to be spatially coherent

each pixel is replaced with the mean value of its cluster Source: A. Oliva 26

slide-27
SLIDE 27

K-means using color alone (k=11 clusters) Showing 4 of the segments, (not necessarily connected) Some are good, some meaningless

Source: A. Oliva 27

slide-28
SLIDE 28

Including spatial relationships

⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ = y x v u Y z

color coordinates (or r,g,b) spatial coordinates

Augment data to be clustered with spatial coordinates.

Source: A. Oliva 28

slide-29
SLIDE 29
  • Clustering based on (r,g,b,x,y) values

enforces more spatial coherence

K-means using color and position, 20 segments

Still misses goal of perceptually pleasing or useful segmentation No measure of texture Hard to pick K…

Source: A. Oliva

29

slide-30
SLIDE 30

K-Means for segmentation

  • Pros

– Very simple method – Converges to a local minimum of the error function

  • Cons

– Memory-intensive – Need to pick K – Sensitive to initialization – Sensitive to outliers – Parametric

Source: S. Lazebnik

30

slide-31
SLIDE 31

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

Method 2: Mean shift clustering

  • A versatile technique for clustering-based

segmentation

  • D. Comaniciu and P

. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.

Source: A. Oliva 31

slide-32
SLIDE 32

The mean shift algorithm seeks modes or local maxima of density in the feature space

Mean shift mode-finding algorithm

image Feature space (L*u*v* color values)

32

slide-33
SLIDE 33

Search
 window Center of mass Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel 33

slide-34
SLIDE 34

Search
 window Center of mass Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel 34

slide-35
SLIDE 35

Search
 window Center of mass Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel 35

slide-36
SLIDE 36

Search
 window Center of mass Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel 36

slide-37
SLIDE 37

Search
 window Center of mass Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel 37

slide-38
SLIDE 38

Search
 window Center of mass Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel 38

slide-39
SLIDE 39

Search
 window Center of mass

Slide by Y. Ukrainitz & B. Sarel 39

slide-40
SLIDE 40
  • Cluster: all data points in the attraction

basin of a mode

Mean shift clustering

Slide by Y. Ukrainitz & B. Sarel 40

slide-41
SLIDE 41
  • 1. Convert the image into tokens (via color, gradients, texture measures etc).
  • 2. Choose initial search window locations uniformly in the data.
  • 3. Compute the mean shift window location for each initial position.
  • 4. Merge windows that end up on the same “peak” or mode.
  • 5. The data these merged windows traversed are clustered together.

Mean Shift Segmentation

Szeliski – Chapter 5 – Segmentation, 5.3 Mean shift

Pixels in L*u* space

Clustering results after ~160 mean shift procedures Corresponding trajectories with peaks marked as red dots

Source: A. Oliva 41

slide-42
SLIDE 42

Mean Shift Segmentation Results

Source: A. Oliva 42

slide-43
SLIDE 43

Mean shift pros and cons

  • Pros

– Nonparametric (e.g. K-means has a parametric model for cluster shape) – Clusters are places where data points tend to be close together – Just a single parameter (window size) – Finds variable number of modes – Robust to outliers

  • Cons

– Output depends on window size – Computationally expensive – Does not scale well with dimension of feature space

Slide credit: S. Lazebnik

43

slide-44
SLIDE 44

Method 3: Graph-based Image Segmentation

Build a weighted graph G = (V,E) from image V: image pixels E: connections between pairs of nearby pixels

Segmentation = graph partition

A different way of thinking about segmentation…

Source: A. Oliva 44

slide-45
SLIDE 45
  • Fully connected graph (node for every pixel i,j)
  • Edge/link between every pair of pixels: p,q
  • Each edge is weighted by the affinity or similarity of the two nodes:

– cost cpq for each link: cpq measures similarity (or affinity) – similarity is inversely proportional to difference in color and position

wij i j

Graph formulation

q p Cpq

Source: A. Oliva 45

slide-46
SLIDE 46

Segmentation by graph cut

  • Break Graph into Segments

– Delete links that cross between segments – Easiest to break links that have low cost (similarity or affinity)

  • similar pixels should be in the same segments
  • dissimilar pixels should be in different segments

A B C

Source: S. Seitz

wij i j

46

slide-47
SLIDE 47

Graph-based Image Segmentation

1- Get vectors of data 2b- Build a similarity/affinity matrix 2a- Build a similarity graph 3- Calculate eigenvectors 4- Cut the graph: apply threshold to eigenvectors

Goal: Given data points X1, …, Xn and similarities w(Xi,Xj), partition the data into groups so that points in a group are similar and points in different groups are dissimilar.

Source: A. Oliva 47

slide-48
SLIDE 48

1- Vectors of data

We represent each pixel by a feature vector x, and define a distance function appropriate for this feature representation (e.g. euclidean distance). Features can be brightness value, color– RGB, L*u*v; texton histogram, etc- and calculate distances between vectors (e.g. Euclidean distance)

Textons (texture features)

Source: A. Oliva 48

slide-49
SLIDE 49

From distance to affinity

  • We represent each pixel by a feature vector

x, and define a distance function appropriate for this feature representation

  • Then we can convert the distance between

two feature vectors into an affinity/similarity measure with the help of a generalized Gaussian kernel: ⎟ ⎠ ⎞ ⎜ ⎝ ⎛−

2 2

) , ( dist 2 1 exp

j i x

x σ

Source: S. Lazebnik

49

slide-50
SLIDE 50

Graph-based Image Segmentation

1- Get vectors of data 2b- Build a similarity/affinity matrix 2a- Build a similarity graph 3- Calculate eigenvectors 4- Cut the graph: apply threshold to eigenvectors

Goal: Given data points X1, …, Xn and similarities w(Xi,Xj), partition the data into groups so that points in a group are similar and points in different groups are dissimilar.

Source: A. Oliva 50

slide-51
SLIDE 51

a e d c b

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ 1 1 1 1 1 1 1 1

Adjacency Matrix

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

a b c d e a b c d e

  • - - - -
  • - - - -
  • - - - -
  • - - - -
  • - - - -

2a- Graph

51

slide-52
SLIDE 52

2a- Graph

52

a e d c b

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ 1 1 1 1 1 1 1 1

Adjacency Matrix

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

a b c d e a b c d e

0 - - - -

  • 0 - - -
  • - 0 - -
  • - - 0 -
  • - - - 0
slide-53
SLIDE 53

2a- Graph

53

a e d c b

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ 1 1 1 1 1 1 1 1

Adjacency Matrix

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

a b c d e a b c d e

0 1 - - - 1 0 - - -

  • - 0 - -
  • - - 0 -
  • - - - 0
slide-54
SLIDE 54

2a- Graph

54

a e d c b

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ 1 1 1 1 1 1 1 1

Adjacency Matrix

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

a b c d e a b c d e

0 1 - - 1 1 0 - - -

  • - 0 - -
  • - - 0 -

1 - - - 0

slide-55
SLIDE 55

2a- Graph

55

a e d c b

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ 1 1 1 1 1 1 1 1

Adjacency Matrix

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

a b c d e a b c d e

slide-56
SLIDE 56

2a- What is a weighted graph?

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ 1 1 7 . 2 . 1 1 6 . 7 . 6 . 1 4 . 3 . 2 . 4 . 1 1 . 3 . 1 . 1

Affinity Matrix represents the weighted links

See Forsyth-Ponce chapter

a e d c b 6

W =

region same the to belong j & i y that probabilit :

ij

W

a b c d e a b c d e Diagonal: each point with itself is 1 Strong links/edges Weak links/edges No links/edges connected

1 .1 .3 0 0 0 1 .4 0 .2 .3 .4 1 .6 .7 0 0 .6 1 1 0 .2 .7 1 1

i,j are the pixels in the image

Source: A. Oliva 56

slide-57
SLIDE 57
  • Similarity matrix:

Slides from Jianbo Shi 2 2 2 ) ( ) (

,

X j i

X X j i

e w

σ − −

=

Weight matrix associated with the graph (larger values are lighter)

2b- Building Affinity Matrix

57

slide-58
SLIDE 58

Affinity matrix of a natural image

Similarity of image pixels to selected pixel Brighter means more similar N*M pixels N*M pixels M pixels N pixels

58

slide-59
SLIDE 59

Graph-based Image Segmentation

1- Get vectors of data 2b- Build a similarity matrix 2a- Build a similarity graph 3- Calculate eigenvectors 4- Cut the graph: apply threshold to eigenvectors

Goal: Given data points X1, …, Xn and similarities w(Xi,Xj), partition the data into groups so that points in a group are similar and points in different groups are dissimilar.

Source: A. Oliva 59

slide-60
SLIDE 60

Partition a graph with minimum cut

  • Cut: sum of the weight of the cut edges:
  • Natural idea: minimum cut

Source: Jianbo Shi

60

slide-61
SLIDE 61

Drawbacks of Minimum Cut

  • Weight of cut is directly proportional to

the number of edges in the cut.

Ideal Cut Cuts with lesser weight than the ideal cut

* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

61

slide-62
SLIDE 62

Normalize by the total volume of connections [Shi and Malik 2000]

Normalized Cut is often better

where is the sum

  • f all weights “associated” with A

Source: A. Oliva 62

slide-63
SLIDE 63

Normalized Cut As Generalized Eigenvalue problem

After lots of simplification, can show that this is:

. 1 }, , 1 { with , ) ( ) , ( = − ∈ − = D y b y Dy y y W D y B A Ncut

T i T T

where W is the affinity matrix and D is a diagonal degree matrix.

For detailed derivation: http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf

Source: A. Oliva 63

slide-64
SLIDE 64
  • Use a continuous relaxation!

  • They show that the 2nd smallest eigenvector solution y is an

approximate solution to the original normalized cuts problem. 


  • Choose a quantization threshold that maximizes the criterion

— i.e all components of y above that threshold go to one, all below go to -b

Normalized cuts

http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf

Want to solve integer program:

Binary Binary

Source: A. Oliva 64

slide-65
SLIDE 65

An ideal case

On Spectral Clustering: Analysis and an algorithm. Andrew Y. Ng, Michael I. Jordan, Yair Weiss, NIPS 2001

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Affinity Matrix

W= What are the eigenvectors of this matrix?

65

slide-66
SLIDE 66

An ideal case

On Spectral Clustering: Analysis and an algorithm. Andrew Y. Ng, Michael I. Jordan, Yair Weiss, NIPS 2001

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Affinity Matrix

W= 1 1 1 1 1 1 … Eigenvectors: λ =4 λ =2

66

slide-67
SLIDE 67

An ideal case

On Spectral Clustering: Analysis and an algorithm. Andrew Y. Ng, Michael I. Jordan, Yair Weiss, NIPS 2001

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Affinity Matrix

W= Eigenvectors: 1 1 1 1 1 1 λ =4 λ =2 But we do not know the ordering, so W with have some random permutation:

67

slide-68
SLIDE 68

Eigenvectors and blocks

  • Block weight matrices have block eigenvectors:
  • Near-block matrices have near-block eigenvectors:

1 1 1 1 1 1 1 1 eigensolver .71 .71 .71 .71

λ1= 2 λ2= 2 λ3= 0 λ4= 0

1 1 .2 1 1

  • .2

.2 1 1

  • .2

1 1 eigensolver .71 .69 .14

  • .14

.69 .71

λ1= 2.02 λ2= 2.02 λ3= -0.02 λ4= -0.02

* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University

e1 e2

68

slide-69
SLIDE 69

Spectral Space

Can put items into blocks by eigenvectors:

1 1 .2 1 1

  • .2

.2 1 1

  • .2

1 1 .71 .69 .14

  • .14

.69 .71

e1 e2 e1 e2

1 .2 1 .2 1 1 1 1

  • .2

1

  • .2

1 .71 .14 .69 .69

  • .14

.71

e1 e2 e1 e2

* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University

Clusters clear regardless of row ordering:

69

slide-70
SLIDE 70

The eigenvectors correspond the 2nd smallest to the 9th smallest eigenvalues

70

slide-71
SLIDE 71

Many different spectral clustering methods…

1- Get vectors of data 2- Build a similarity graph 3- Calculate eigenvectors 4- Apply threshold to largest eigenvectors

Goal: Given data points X1, …, Xn and similarities w(Xi,Xj), partition the data into groups so that points in a group are similar and points in different groups are dissimilar.

1- Get vectors of data 2- Build normalized cost matrix 3- Get eigenvectors with smallest eigenvalues 4- Apply threshold

Shi & Malik

Source: A. Oliva 71

slide-72
SLIDE 72

Source: A. Oliva 72

slide-73
SLIDE 73

The Eigenvectors

Source: A. Oliva 73

slide-74
SLIDE 74

Normalized cut

Source: A. Oliva 74

slide-75
SLIDE 75

Normalized cut

Source: A. Oliva 75

slide-76
SLIDE 76

Normalized cut

Source: A. Oliva 76

slide-77
SLIDE 77

Application: selective search

[Uijlings et al., "Selective Search for Object Recognition”, 2013]

77

slide-78
SLIDE 78

78

slide-79
SLIDE 79

Min-cut for user-in-the-loop segmentation

[Rother et al., “GrabCut”, 2004] User input Min-cut segmentation Solution after iteration

  • Formulation makes it easy to add local evidence
  • How much does this pixel look like the foreground?

  • Formulate as probabilistic graphical model.

79

slide-80
SLIDE 80

Min-cut for user-in-the-loop segmentation

[Rother et al., “GrabCut”, 2004]

80

slide-81
SLIDE 81

What about edges?

81

slide-82
SLIDE 82

Image filters not “edge-aware”

Gaussian filter

[Paris et al. “A Gentle Introduction to Bilateral Filtering and its Applications”, 2008]

82

slide-83
SLIDE 83

Bilateral filter

[Paris et al. “A Gentle Introduction to Bilateral Filtering and its Applications”, 2008]

Gaussian filter Bilateral filter

83

slide-84
SLIDE 84

Bilateral filter

[Paris et al. “A Gentle Introduction to Bilateral Filtering and its Applications”, 2008]

84

slide-85
SLIDE 85

Bilateral filter

[Paris et al. “A Gentle Introduction to Bilateral Filtering and its Applications”, 2008]

85

slide-86
SLIDE 86

Bilateral filter + edge detection

[Paris et al. “A Gentle Introduction to Bilateral Filtering and its Applications”, 2008] Without bilateral With bilateral

86

slide-87
SLIDE 87

Neural nets that operate in “bilateral space”

[Gharbi et al. Deep Bilateral Learning for Real-Time Image Enhancement”, 2017]

87

slide-88
SLIDE 88

Next lecture: recent advances

88