Integrating Vision and Haptics for Object Recognition Sibel Toprak - - PowerPoint PPT Presentation

▶
integrating vision and haptics for object recognition
SMART_READER_LITE
LIVE PREVIEW

Integrating Vision and Haptics for Object Recognition Sibel Toprak - - PowerPoint PPT Presentation

Jan 06, 2024 343 likes •599 views

Integrating Vision and Haptics for Object Recognition Sibel Toprak Seminar Talk in Intelligent Robotics November 9, 2015 Motivation Robust object recognition capabilities required in most robot applications However: Object

slide-1
SLIDE 1

Integrating Vision and Haptics for Object Recognition

Sibel Toprak Seminar Talk in Intelligent Robotics November 9, 2015

slide-2
SLIDE 2

Motivation

  • Robust object

recognition capabilities required in most robot applications However:

  • Object recognition

based on vision alone not reliable enough in most cases

2 http://japan-wa.de/roboter-hotel/

slide-3
SLIDE 3

Visual Object Recognition

Problem 1: Not all discerning features are visual! Problem 2: The sight is not always perfect!

‟Robot, get me the full bottle!” Which one is it? Occlusions and darkness in the scene, objects that are particularly visually complex, …

3 https://www.sigg.com, http://www.asdirect.fr/images/produits/p6057-2.jpg

slide-4
SLIDE 4

Haptic Exploration Procedures

4

[Lederman and Klatzky, 2009; 1987]

slide-5
SLIDE 5

Haptic Object Recognition

Submodalities: Advantages for: Object Learning More rich and distinctive characterization of objects during exploration Object Manipulation Learning how to interact with

  • bjects based on the received

haptic feedback

5

tactile and kinesthetic (e.g. texture and weight of objects)

http://de.123rf.com/photo_3716245_kleines-kind-mit-apfel--8-isoliert-auf-wei.html

slide-6
SLIDE 6

Question

Classification Output Visual Input ⁞ ⁞ Haptic Input ⁞ ⁞

Integration Strategies?

6

slide-7
SLIDE 7

Question

Classification Output Visual Input ⁞ ⁞ Haptic Input ⁞ ⁞ Level of Integration Weighting Neural Network Architecture

7

Not covered here!

slide-8
SLIDE 8

General Approach

8

Validate the different integration strategies using neuroscientific evidence

https://commons.wikimedia.org/wiki/File:Human-brain.SVG

slide-9
SLIDE 9

Sensory-Level Integration

Classification Output Visual Input ⁞ ⁞ Haptic Input ⁞ ⁞ Insert any established Artificial Neural Network Architecture here e.g. Deep Learning Architectures, Echo State Network, …

9

slide-10
SLIDE 10

Issues

  • No modularity

⇒Difficult reuse of processing results for other tasks ⇒Not much insight into what is actually learned ⇒Waste of computational effort

  • Neuroscientific counterevidence

⇒Next slide

10

slide-11
SLIDE 11

Cortical Sensory Areas in the Brain

11 http://www.d.umn.edu/~jfitzake/Lectures/DMED/SensoryPhysiology/GeneralPrinciples/CodingTheories.html

slide-12
SLIDE 12

Decision-Level Integration

Classification Output Visual Input ⁞ ⁞ Haptic Input ⁞ ⁞

Visual Object Recognition Module Haptic Object Recognition Module Decision-Level Integration Module

12

slide-13
SLIDE 13

Issues

  • Dominance of vision over haptics

⇒Result of haptic part might not be needed ⇒Waste of computational effort

  • Recent neuroscientific counterevidence

⇒Next slides

13

slide-14
SLIDE 14

Somatosensory Cortex

14 https://classconnection.s3.amazonaws.com/698/flashcards/723698/png/untitled1319491580334.png

Brodmann Areas Sensory Homunculus Hierarchical processing of somatosensory information Somatotopic

  • rganization
slide-15
SLIDE 15

Visual Cortex

15

Striate Area (V1) Hierarchical processing of visual information

http://people.brandeis.edu/~teuber/Vision.pdf

Extrastriate Areas (V2-V5) Retinotopic organization

slide-16
SLIDE 16

Ventral and Dorsal Streams

16 https://commons.wikimedia.org/wiki/File:Ventral-dorsal_streams.svg

Ventral Stream: (aka "what pathway") Object Recognition Processing Stream Primary Visual (Striate) Cortex Dorsal Stream: (aka "where/ how pathway") Motion/ Object Location Processing Stream !

[Goodale and Milner, 1992; Ungerleider and Mishkin, 1982]

slide-17
SLIDE 17

Lateral Occipital Complex

17

[Lacey et al., 2009; Amedi et al., 2001]

Lateral View Ventral View Object-related regions in the visual and haptic modalities Subregion of LOC responds selectively to objects in both vision and touch

slide-18
SLIDE 18

Lateral Occipital Complex

18 http://cercor.oxfordjournals.org/content/12/11/1202/F2.full

[Amedi et al., 2002]

Object-related regions in the visual, tactile and auditory modalities No auditory object- related activation

  • bserved in LOC

LOC involved in the recovery of geometrical shape of objects

slide-19
SLIDE 19

Interim Conclusion

Classification Output Visual Input ⁞ ⁞ Haptic Input ⁞ ⁞

Hierarchical Visual Feature Extraction Hierarchical Haptic Feature Extraction Visuo-Haptic Geometrical Shape Recovery

19 Unimodal Shape Information

Lateral Occipital Complex Visual Cortex Somatosensory Cortex

slide-20
SLIDE 20

Open Questions

Question 1:

  • LOC at a pretty early stage
  • f the ventral pathway
  • Apparently not responsible

for object recognition per se ⇒ Basically: When and how does object recognition actually occur?

20 https://commons.wikimedia.org/wiki/File:Ventral-dorsal_streams.svg

slide-21
SLIDE 21

Open Questions

Question 2:

  • Further visuo-haptically

perceivable object properties: Texture and volume/ size ⇒ No integration?

  • Only haptically perceivable:

Hardness, temperature and weight

  • Only visually perceivable:

Color, … ⇒ (Any) role in object recogn.?

21

[Lederman and Klatzky, 2009]

slide-22
SLIDE 22

Feature-Level Integration

Classification Output Visual Input ⁞ ⁞ Haptic Input ⁞ ⁞

Hierarchical Visual Feature Extraction Hierarchical Haptic Feature Extraction Visuo-Haptic Geometrical Shape Recovery

22 Unimodal Shape Information

Lateral Occipital Complex

Feature Integration

???

Shape Color, …

Visual Cortex Somatosensory Cortex

slide-23
SLIDE 23

Summary

Question: How to integrate vision and haptics for more robust object recognition? Possible Integration Strategies: (1) Sensory-Level (2) Decision-Level (3) Feature-Level Conclusion: (3) Good starting point to investigate the importance of shape in object recognition

23

slide-24
SLIDE 24

References

  • Lederman, Susan J., and Roberta L. Klatzky. "Haptic perception: A

tutorial."Attention, Perception, & Psychophysics 71.7 (2009): 1439-1459.

  • Lederman, Susan J., and Roberta L. Klatzky. "Hand movements: A window into

haptic object recognition." Cognitive psychology 19.3 (1987): 342-368.

  • Amedi, Amir, et al. "Functional imaging of human crossmodal identification and
  • bject recognition." Experimental Brain Research 166.3-4 (2005): 559-571.
  • Goodale, Melvyn A., and A. David Milner. "Separate visual pathways for perception

and action." Trends in neurosciences 15.1 (1992): 20-25.

  • Mishkin, Mortimer, Leslie G. Ungerleider, and Kathleen A. Macko. "Object vision

and spatial vision: two cortical pathways." Trends in neurosciences 6 (1983): 414- 417.

  • Lacey, Simon, et al. "A putative model of multisensory object representation."

Brain topography 21.3-4 (2009): 269-274.

  • Amedi, Amir, et al. "Visuo-haptic object-related activation in the ventral visual

pathway." Nature neuroscience 4.3 (2001): 324-330.

  • Amedi, Amir, et al. "Convergence of visual and tactile shape processing in the

human lateral occipital complex." Cerebral Cortex 12.11 (2002): 1202-1212.

  • Ungerleider, Leslie G., and Luiz Pessoa. "What and where pathways." Scholarpedia

3.11 (2008): 5342.

24