Semantic Segmentation Lecture: Segmentation Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab 17-Oct-2019 1 St Stanfor ord University
CS 131 Roadmap Semantic Segmentation Pixels Segments Images Videos Web Neural networks Convolutions Recognition Resizing Motion Convolutional Edges Detection Segmentation Tracking neural networks Descriptors Machine learning Clustering 17-Oct-2019 2 St Stanfor ord University
What we will learn today • Introduction to segmentation and clustering Semantic Segmentation • Gestalt theory for perceptual grouping • Agglomerative clustering • Oversegmentation 17-Oct-2019 3 Reading: [FP] Chapters: 14.2, 14.4 St Stanfor ord University
Semantic Segmentation 17-Oct-2019 4 ord University Stanfor St
Image Segmentation • Goal: identify groups of pixels that go together Semantic Segmentation 17-Oct-2019 5 St Stanfor ord University Slide credit: Steve Seitz, Kristen Grauman
The Goals of Segmentation • Separate image into coherent “objects” Semantic Segmentation Image Human segmentation 17-Oct-2019 6 St Stanfor ord University Slide credit: Svetlana Lazebnik
The Goals of Segmentation • Separate image into coherent “objects” Semantic Segmentation • Group together similar-looking pixels for efficiency of further processing Slide credit: Svetlana Lazebnik 17-Oct-2019 “superpixels” 7 X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003. St Stanfor ord University
Segmentation for feature support Semantic Segmentation 50x50 Patch 50x50 Patch 17-Oct-2019 8 St Stanfor ord University Slide: Derek Hoiem
Segmentation for efficiency Semantic Segmentation [Felzenszwalb and Huttenlocher 2004] 17-Oct-2019 [Hoiem et al. 2005, Mori 2005] [Shi and Malik 2001] 9 St Stanfor ord University Slide: Derek Hoiem
Segmentation as a result Semantic Segmentation 17-Oct-2019 10 Rother et al. 2004 St Stanfor ord University
Types of segmentations Semantic Segmentation Oversegmentation Undersegmentation 17-Oct-2019 Multiple Segmentations 11 St Stanfor ord University
One way to think about “segmentation” is Clustering Semantic Segmentation Clustering: group together similar data points and represent them with a single token Key Challenges: 1) What makes two points/images/patches similar? 17-Oct-2019 2) How do we compute an overall grouping from pairwise similarities? 12 St Stanfor ord University Slide: Derek Hoiem
Why do we cluster? • Summarizing data – Look at large amounts of data Semantic Segmentation – Patch-based compression or denoising – Represent a large continuous vector with the cluster number • Counting – Histograms of texture, color, SIFT vectors • Segmentation 17-Oct-2019 – Separate the image into different regions • Prediction – Images in the same cluster may have the same labels 13 Stanfor St ord University Slide: Derek Hoiem
How do we cluster? • Agglomerative clustering Semantic Segmentation – Start with each point as its own cluster and iteratively merge the closest clusters • K-means (next lecture) – Iteratively re-assign points to the nearest cluster center • Mean-shift clustering (next lecture) – Estimate modes of pdf 17-Oct-2019 14 St Stanfor ord University
General ideas • Tokens Semantic Segmentation – whatever we need to group (pixels, points, surface elements, etc., etc.) • Bottom up clustering – tokens belong together because they are locally coherent • Top down clustering – tokens belong together because they lie on the same visual entity 17-Oct-2019 (object, scene…) > These two are not mutually exclusive 15 St Stanfor ord University
Examples of Grouping in Vision Semantic Segmentation Grouping video frames into shots Determining image regions 17-Oct-2019 What things should Figure-ground be grouped? What cues indicate groups? Object-level grouping 16 St Stanfor ord University Slide credit: Kristen Grauman
Similarity Semantic Segmentation 17-Oct-2019 17 St Stanfor ord University Slide credit: Kristen Grauman
Symmetry Semantic Segmentation 17-Oct-2019 18 Slide credit: Kristen Grauman St Stanfor ord University
Common Fate Semantic Segmentation 17-Oct-2019 Image credit: Arthus-Bertrand (via F. Durand) 19 Slide credit: Kristen Grauman St Stanfor ord University
Proximity Semantic Segmentation 17-Oct-2019 Slide credit: Kristen Grauman 20 St Stanfor ord University
Muller-Lyer Illusion Semantic Segmentation • What makes the bottom line look longer than the top line? 17-Oct-2019 21 St Stanfor ord University
What we will learn today • Introduction to segmentation and clustering Semantic Segmentation • Gestalt theory for perceptual grouping • Agglomerative clustering • Oversegmentation 17-Oct-2019 22 St Stanfor ord University
The Gestalt School • Grouping is key to visual perception • Elements in a collection can have properties that Semantic Segmentation result from relationships – “The whole is greater than the sum of its parts” Illusory/subjective Occlusion contours Slide credit: Svetlana Lazebnik 17-Oct-2019 Familiar configuration 23 http://en.wikipedia.org/wiki/Gestalt_psychology Stanfor St ord University
Gestalt Theory • Gestalt: whole or group Semantic Segmentation – Whole is greater than sum of its parts – Relationships among parts can yield new properties/features • Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) “I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees.” 17-Oct-2019 Max Wertheimer (1880-1943) Untersuchungen zur Lehre von der Gestalt, Psychologische Forschung , Vol. 4, pp. 301-350, 1923 http://psy.ed.asu.edu/~classics/Wertheimer/Forms/forms.htm 24 Stanfor St ord University
Gestalt Factors Semantic Segmentation 17-Oct-2019 Image source: Forsyth & Ponce • These factors make intuitive sense, but are very difficult to translate into algorithms. 25 St Stanfor ord University
Continuity through Occlusion Cues Semantic Segmentation 17-Oct-2019 26 St Stanfor ord University
Continuity through Occlusion Cues Semantic Segmentation 17-Oct-2019 Continuity, explanation by occlusion 27 St Stanfor ord University
Continuity through Occlusion Cues Semantic Segmentation Image source: Forsyth & Ponce 17-Oct-2019 28 St Stanfor ord University
Continuity through Occlusion Cues Semantic Segmentation Image source: Forsyth & Ponce 17-Oct-2019 29 St Stanfor ord University
Figure-Ground Discrimination Semantic Segmentation 17-Oct-2019 30 St Stanfor ord University
The Ultimate Gestalt? Semantic Segmentation 17-Oct-2019 31 St Stanfor ord University
What we will learn today • Introduction to segmentation and clustering Semantic Segmentation • Gestalt theory for perceptual grouping • Agglomerative clustering • Oversegmentation 17-Oct-2019 32 St Stanfor ord University
Clustering: distance measure Clustering is an unsupervised learning method. Given items Semantic Segmentation , the goal is to group them into clusters. We need a pairwise distance/similarity function between items, and sometimes the desired number of clusters. When data (e.g. images, objects, documents) are represented by feature vectors, commonly used measures are: - Euclidean distance. 17-Oct-2019 - cosine similarity. 34 St Stanfor ord University
Defining Distance Measures Let x and x’ be two objects from the universe of possible objects. Semantic Segmentation The distance (similarity) between x and x’ is a real number denoted by sim(x, x’). • The Euclidean distance is defined as 𝑒𝑗𝑡𝑢 𝑦, 𝑦 ' = ∑ 𝑦 * − 𝑦′ * - • In contrast, the cosine similarity measure would be 17-Oct-2019 35 St Stanfor ord University
Desirable Properties of a Clustering Algorithms • Scalability (in terms of both time and space) Semantic Segmentation • Ability to deal with different data types • Minimal requirements for domain knowledge to determine input parameters • Interpretability and usability Optional – Incorporation of user-specified constraints 17-Oct-2019 36 St Stanfor ord University
Animated example Semantic Segmentation 17-Oct-2019 source 37 St Stanfor ord University
Animated example Semantic Segmentation 17-Oct-2019 source 38 St Stanfor ord University
Animated example Semantic Segmentation 17-Oct-2019 source 39 St Stanfor ord University
Agglomerative clustering Semantic Segmentation 17-Oct-2019 Slide credit: Andrew Moore 40 St Stanfor ord University
Agglomerative clustering Semantic Segmentation 17-Oct-2019 Slide credit: Andrew Moore 41 St Stanfor ord University
Agglomerative clustering Semantic Segmentation 17-Oct-2019 Slide credit: Andrew Moore 42 St Stanfor ord University
Agglomerative clustering Semantic Segmentation 17-Oct-2019 Slide credit: Andrew Moore 43 St Stanfor ord University
Recommend
More recommend
