Scene Understanding Introduction & Overview Outline Motivation - PowerPoint PPT Presentation

Scene Understanding Introduction & Overview

Outline • Motivation • The problems • Scene perception • Approaches

Objects in context

Objects in context

Συνεργός

Scene classification • Natural / Urban • Sunny / Cloudy • Open / Closed • Object content • Location • …

Spatial boundary

3D scene understanding (from a single image)

3D scene understanding (from a single image)

3D scene understanding (from a single image)

Image memorability • We see hundreds of images a day • We even remember some of them (but forget many others) • Subjective? Not completely • “ not an inexplicable phenomenon”

Image memorability

Image memorability • Understanding and Predicting Image Memorability at a Large Scale (ICCV 2015)

Scene perception • 1969: Can remember and describe images seen for 0.1 seconds • 1981: Three levels of representation 1. Prominent object 2. Multiple-component 3. Global scene-emergent features • Object-centered vs. Space-centered

Time frame of scene perception • Global-to-local analysis • 20-30 ms: Natural/Urban, Open/Closed • 100ms: Basic category (beach, forest, etc.) • 200ms+: Object identities

Temporal context • Classification of ambiguous images depend on previously perceived scenes

GIST descriptor • Meaningful information from a glimpse • “naturalness, openness, roughness, expansion, ruggedness”

GIST descriptor • Segment image by 4x4 grid, compute oriented histograms

3D Scene from single images • Geometric classes • Support, Vertical, Sky • Vertical subclasses: Left, center, right Porous, solid

3D Scene from single images • Divide image into “superpixels” • Group superpixels into larger regions • Compute features for each region • Compute confidence levels for different geometric classes

Photo pop-up • Find connected components for vertical regions • Fit line segments to base of each component • Compute depth at each point

References Barenholtz, E. (2013). Quantifying the role of context in visual object recognition. Visual Cognition, 22(1), 30-56. • Biederman, I. (1981). On the semantics of a glance at a scene. In M. Kubovy & J. R. Pomerantz (Eds.), Perceptual organization (pp. 213–263). • Hillsdale, NJ: Lawrence Erlbaum Associates. Fei-Fei, L., Iyer, A., Koch, C., & Perona, P . (2007). What do we perceive in a glance of a real-world scene? Journal of Vision, 7(1), 1–29. • Isola, P ., et al. (2011). Understanding the intrinsic memorability of images. Advances in Neural Information Processing Systems, 24. • Jahangiri, E., et al. (2014). Object-Level Generative Models for 3D Scene Understanding. SUNw: Scene Understanding Workshop. • Khosla, A., Raju, A. S., Torralba, A., & Oliva, A. (2015). Understanding and Predicting Image Memorability at a Large Scale. 2015 IEEE • International Conference on Computer Vision (ICCV). Oliva, A., & Torralba, A. (2001). Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of • Computer Vision, 42, 145–175. Oliva, A., & Torralba, A. (2006). Building the gist of a scene: The role of global image features in recognition. Progress in Brain Research • Visual Perception - Fundamentals of Awareness: Multi-Sensory Integration and High-Order Perception, 23-36. Oliva, A., & Torralba, A. (2007). The role of context in object recognition. Trends in Cognitive Sciences, 11(12), 520-527. • Oliva, A. (2014). Scene Perception. In L. M. Chalupa & J. S. Warner (Eds.), The New Visual Neurosciences (pp. 725-732). Cambridge, MA: MIT • Press. Park, S., et al. (2011). Disentangling Scene Content from Spatial Boundary: Complementary Roles for the Parahippocampal Place Area and • Lateral Occipital Complex in Representing Real-World Scenes. Journal of Neuroscience, 31(4), 1333-1340. Potter, M. C., & Levy, E. I. (1969). Recognition memory for a rapid sequence of pictures. Journal of Experimental Psychology, 81, 10–15. •