Scene Understanding Introduction & Overview Outline Motivation - - PowerPoint PPT Presentation

Scene Understanding

Introduction & Overview

Outline

• Motivation
• The problems
• Scene perception
• Approaches

Objects in context

Objects in context

Συνεργός

Scene classification

• Natural / Urban
• Sunny / Cloudy
• Open / Closed
• Object content
• Location
• …

Spatial boundary

3D scene understanding (from a single image)

3D scene understanding (from a single image)

3D scene understanding (from a single image)

Image memorability

• We see hundreds of images a day
• We even remember some of them

(but forget many others)

• Subjective? Not completely
• “not an inexplicable phenomenon”

Image memorability

Image memorability

• Understanding and Predicting

Image Memorability at a Large Scale (ICCV 2015)

Scene perception

• 1969: Can remember and describe images seen for 0.1 seconds
• 1981: Three levels of representation
• 1. Prominent object
• 2. Multiple-component
• 3. Global scene-emergent features
• Object-centered
• vs. Space-centered

Time frame of scene perception

• Global-to-local analysis
• 20-30 ms: Natural/Urban, Open/Closed
• 100ms: Basic category (beach, forest, etc.)
• 200ms+: Object identities

Temporal context

• Classification of ambiguous images depend on previously perceived scenes

GIST descriptor

• Meaningful information from a glimpse
• “naturalness, openness, roughness, expansion, ruggedness”

GIST descriptor

• Segment image by 4x4 grid, compute oriented histograms

3D Scene from single images

• Geometric classes
• Support, Vertical, Sky
• Vertical subclasses: Left, center, right

Porous, solid

3D Scene from single images

• Divide image into “superpixels”
• Group superpixels into larger regions
• Compute features for each region
• Compute confidence levels for different geometric classes

Photo pop-up

• Find connected components for vertical regions
• Fit line segments to base of each component
• Compute depth at each point

References

• Barenholtz, E. (2013). Quantifying the role of context in visual object recognition. Visual Cognition, 22(1), 30-56.
• Biederman, I. (1981). On the semantics of a glance at a scene. In M. Kubovy & J. R. Pomerantz (Eds.), Perceptual organization (pp. 213–263).

Hillsdale, NJ: Lawrence Erlbaum Associates.

• Fei-Fei, L., Iyer, A., Koch, C., & Perona, P

. (2007). What do we perceive in a glance of a real-world scene? Journal of Vision, 7(1), 1–29.

• Isola, P

., et al. (2011). Understanding the intrinsic memorability of images. Advances in Neural Information Processing Systems, 24.

• Jahangiri, E., et al. (2014). Object-Level Generative Models for 3D Scene Understanding. SUNw: Scene Understanding Workshop.
• Khosla, A., Raju, A. S., Torralba, A., & Oliva, A. (2015). Understanding and Predicting Image Memorability at a Large Scale. 2015 IEEE

International Conference on Computer Vision (ICCV).

• Oliva, A., & Torralba, A. (2001). Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of

Computer Vision, 42, 145–175.

• Oliva, A., & Torralba, A. (2006). Building the gist of a scene: The role of global image features in recognition. Progress in Brain Research

Visual Perception - Fundamentals of Awareness: Multi-Sensory Integration and High-Order Perception, 23-36.

• Oliva, A., & Torralba, A. (2007). The role of context in object recognition. Trends in Cognitive Sciences, 11(12), 520-527.
• Oliva, A. (2014). Scene Perception. In L. M. Chalupa & J. S. Warner (Eds.), The New Visual Neurosciences (pp. 725-732). Cambridge, MA: MIT

Press.

• Park, S., et al. (2011). Disentangling Scene Content from Spatial Boundary: Complementary Roles for the Parahippocampal Place Area and

Lateral Occipital Complex in Representing Real-World Scenes. Journal of Neuroscience, 31(4), 1333-1340.

• Potter, M. C., & Levy, E. I. (1969). Recognition memory for a rapid sequence of pictures. Journal of Experimental Psychology, 81, 10–15.