Perception of Affordances Perception of Affordances Final Status of - - PowerPoint PPT Presentation

▶
perception of affordances perception of affordances
SMART_READER_LITE
LIVE PREVIEW

Perception of Affordances Perception of Affordances Final Status of - - PowerPoint PPT Presentation

Oct 16, 2023 200 likes •496 views

Perception of Affordances Perception of Affordances Final Status of Work Final Status of Work Lucas Paletta & Gerald Fritz Lucas Paletta & Gerald Fritz Computational Perception Group Computational Perception Group Institute of

slide-1
SLIDE 1

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Perception of Affordances Perception of Affordances

Final Status of Work Final Status of Work

Lucas Paletta & Gerald Fritz Lucas Paletta & Gerald Fritz

Computational Perception Group Computational Perception Group Institute of Digital Image Processing Institute of Digital Image Processing JOANNEUM RESEARCH JOANNEUM RESEARCH Forschungsgesellschaft Forschungsgesellschaft mbH mbH Graz, Austria Graz, Austria

slide-2
SLIDE 2

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Implementation of Perception Module Implementation of Perception Module

slide-3
SLIDE 3

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Feature Feature detectors detectors

Perception Module Perception Module

Architecture Architecture

Behaviors Behaviors Learning Learning of

  • f affordances

affordances Execution Execution Control Control

slide-4
SLIDE 4

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

JR Sensor Interface JR Computational Units JR Data Wrapper JR Control

Points of Integration Points of Integration

slide-5
SLIDE 5

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

read read

( (EntityTrajectoryId EntityTrajectoryId, , TimeStamp TimeStamp) ) ← ← ( (EntityFrame EntityFrame) )

CU #1 CU #2 CU #N

…

Computational Computational Perception Perception Toolbox (CPT) Toolbox (CPT)

real sensors virtual sensors

Sensor Toolbox (ST) Sensor Toolbox (ST)

entity trajectory structure 1 entity trajectory structure N . . .

Entity Entity Trajectory Trajectory Cache Cache (ETC) (ETC) Entity Entity Frame Monitor Frame Monitor (EFM) (EFM)

configEFM configEFM

( (EntityTrajectoryIdList EntityTrajectoryIdList, , EntityFrameAttributeList EntityFrameAttributeList) )

PERCEPTION MODULE (PM) PERCEPTION MODULE (PM)

Entity Entity Structure Structure Generator Module Generator Module (ESGM) (ESGM)

request request

( (EntityTrajectoryId EntityTrajectoryId, , ComputationalUnitId ComputationalUnitId, , ParamList ParamList, , SampleRate SampleRate, , CacheSize CacheSize) ) ← ←( (EntitytTrajectoryId EntitytTrajectoryId, , ComputationalUnitInstId ComputationalUnitInstId) )

configCU configCU

( (ComputationalUnitInstId ComputationalUnitInstId, , ParamList ParamList) )

Perception Module Perception Module

Components Components

slide-6
SLIDE 6

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Perception Module Perception Module

Entity Trajectories Entity Trajectories

ComputationalUnitId

CU #1

Computational Computational Perception Perception Toolbox (CPT) Toolbox (CPT)

ComputationalUnitInst InstId

CU #1

entity trajectory structure ET#1 EntityTrajectoryId ET#2

slide-7
SLIDE 7

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Vision based Affordance Cueing Vision based Affordance Cueing

slide-8
SLIDE 8

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Affordance Cueing Affordance Cueing

2D Affordance Cueing 2D Affordance Cueing

sit! sit! sit! sit!

slide-9
SLIDE 9

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Affordance Cueing Affordance Cueing

Components for Components for Affordance based Recognition Affordance based Recognition

Affordance Monitoring Affordance Application Sensory-Motor Interaction Sensory-Motor Interaction BEHAVIOR BEHAVIOR

FEATURE RECOGNITION

AFFORDANCE ATTENTION AFFORDANCE ATTENTION (Top-down) (Top-down)

Action-Perception Cycle AFFORDANCE RECOGNITION AFFORDANCE RECOGNITION

Final Final State State

OUTCOME OUTCOME CUE CUE AFFORDANCE CUEING AFFORDANCE CUEING

Verify AFFORDANCE AFFORDANCE ENTITY ENTITY support

Paletta et al., ECVP 2006 Paletta et al., ECVP 2006

slide-10
SLIDE 10

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Affordance Cueing Affordance Cueing

Scenario Scenario

slide-11
SLIDE 11

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

entity entity attribute attribute configuration configuration

colour SIFT category ratio L/W G T/B LIFTABLE LIFTABLE NOT LIFTABLE NOT LIFTABLE R M R Y B Bl Gr R R C C R R R N L L L L P P P L T T T T B B B N Y Y Y Y N N N N Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y Y Y N N

Affordance Cueing Affordance Cueing

Affordance Feature Matrix Affordance Feature Matrix

slide-12
SLIDE 12

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Affordance Cueing Affordance Cueing

Hierarchical Hierarchical Structure Structure

visual visual Information Information

colour

  • rientation

depth motion …

point feature segmentation segmentation region region feature grouping grouping

histogram SIFT

  • app. pattern

surface part motion region

grouping grouping feature

histogram

  • geometr. SIFT
  • app. pattern

surface graph motion groups

  • bject
  • bject

modelling modelling

  • bject
  • bject

feature

aspect graph manifold

… … … scene scene modelling modelling

  • bj. conf.

model

  • bject
  • bject

configuration

  • Affordance perception on

all all levels levels of representation possible possible

  • Predefined policies

previously learned previously learned hand-coded hand-coded

  • Any combination from learning

gather powerful policies gather powerful policies Gibson‘s view

slide-13
SLIDE 13

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

visual visual Information Information segmentation segmentation region region feature grouping grouping

histogram SIFT

  • app. pattern

surface part motion region

grouping grouping feature

histogram

  • geometr. SIFT
  • app. pattern

surface graph motion groups

  • bject
  • bject

modelling modelling

  • bject
  • bject

feature

aspect graph manifold

… … …

  • Point features:

Point features:

  • Sift/Blob detector

Sift/Blob detector

  • Region features:

Region features:

  • Color histograms

Color histograms

  • SIFT descriptor

SIFT descriptor

  • Grouping features:

Grouping features:

  • Geometric SIFT

Geometric SIFT

  • DT classification

DT classification

  • Image enhancement

Image enhancement

  • Motion & motion segmentation

Motion & motion segmentation

  • Segmentation

Segmentation

  • Watershed + Energy Merging

Watershed + Energy Merging

  • Connected Components

Connected Components

  • Grouping

Grouping

  • Histogram

Histogram

  • Region Classification

Region Classification

  • Tracking (KLT)

Tracking (KLT)

colour

  • rientation

depth motion …

point feature

Affordance Cueing Affordance Cueing

Computational Perception Computational Perception Toolbox Toolbox

slide-14
SLIDE 14

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

  • riginal
  • riginal

KLT track KLT track color ROI color ROI

Affordance Cueing Affordance Cueing

Affordance Cueing: Affordance Cueing: Implementation Implementation

SIFT SIFT

  • class. SIFT
  • class. SIFT

class.

  • class. hist

hist. . affordance affordance

slide-15
SLIDE 15

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

Affordance Cueing Affordance Cueing

Real-World Affordance Cueing Real-World Affordance Cueing

slide-16
SLIDE 16

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

top = T circular = T unknown size > 1426 non liftable liftable liftable size < 1410 non liftable

Y N Y Y N

liftable

pruning pruning

Fritz et al., SAB 2006; IROS 2006 Fritz et al., SAB 2006; IROS 2006

P(Aliftable|circ) ≈ 0.00 P(Anliftable|circ) ≈ 1.00 P(Aliftable|rect,T) ≈ 0.99 P(Anliftable|rect,T) ≈ 0.01

Affordance Cueing Affordance Cueing

Learning the Classification of Learning the Classification of Affordance Cues Affordance Cues

slide-17
SLIDE 17

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

  • Function based object recognition (Stark & Bowyer 94, Rivlin et al. 95)
  • GRUFF (Generic Representation Using Form and Function)
  • Qualitative recognition of 3D parts, spatial relations
  • Mapping to functional primitives & relations between them
  • Pre-defined feature and object representations

Affordance Cueing Affordance Cueing

Related Work Related Work

slide-18
SLIDE 18

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

purposive (Ballard)

selected selected visual visual information information

function based (Stark/Bowyer)

visual visual information information & & function function

reconstructive (Marr) appearance based (Poggio) affordance affordance based based task based

visual visual information information

Affordance Cueing Affordance Cueing

Frameworks for Recognition Frameworks for Recognition

slide-19
SLIDE 19

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

2D-3D Information Fusion 2D-3D Information Fusion for for Multi-Sensor Affordance Cueing Multi-Sensor Affordance Cueing

slide-20
SLIDE 20

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

2D and 3D Affordance Cueing 2D and 3D Affordance Cueing

Cueing Cueing from Range Images from Range Images

  • Complementary 3D information
  • Learning surface features
  • Bayesian decision fusion for

cueing

planar curved

slide-21
SLIDE 21

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

2D and 3D Affordance Cueing 2D and 3D Affordance Cueing

Calibration between Calibration between 2D and 3D Sensing Devices 2D and 3D Sensing Devices

scenario scenario left camera view left camera view right camera view right camera view laser reflection view laser reflection view

slide-22
SLIDE 22

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

2D and 3D Affordance Cueing 2D and 3D Affordance Cueing

Methodology for Methodology for 3D Feature Extraction 3D Feature Extraction

ROI ROI Detection Detection 2D 2D → → 3D 3D Orientation Orientation Histogram Histogram Entropy Entropy Structure Structure Classification Classification

slide-23
SLIDE 23

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

2D and 3D Affordance Cueing 2D and 3D Affordance Cueing

3D Feature Extraction: 3D Feature Extraction: Results Results

entropy based entropy based structure classification structure classification 2D-3D 2D-3D mapping mapping Gaussian Gaussian filtering filtering

slide-24
SLIDE 24

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

2D and 3D Affordance Cueing 2D and 3D Affordance Cueing

3D Feature Extraction: 3D Feature Extraction: Results Results

curved curved planar planar curved curved

slide-25
SLIDE 25

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

2D and 3D Affordance Cueing 2D and 3D Affordance Cueing

Methodology for Methodology for Multi-Sensor Affordance Cueing Multi-Sensor Affordance Cueing

2D 2D Features Features 3D 3D Features Features 2D 2D Affordance Affordance Classifier Classifier 3D 3D Affordance Affordance Classifier Classifier Information Information Fusion Fusion 2D 2D Affordance Affordance Hypotheses Hypotheses 3D 3D Affordance Affordance Hypotheses Hypotheses

slide-26
SLIDE 26

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

  • Extension to Classical Work on Function based Recognition
  • Methodology for 2D Affordance based Cueing
  • Methodology for Learning and Cueing of Affordances

⇒ ⇒ Rapid Generation of Affordance Hypotheses

Rapid Generation of Affordance Hypotheses

⇒ ⇒ Affordance Cues Need not to be

Affordance Cues Need not to be Pre-defined Pre-defined

  • Implementation within Affordance based Control Architecture
  • Implementation in Real World Environment
  • Methodology for 3D based Features for Affordance Cueing
  • Concept of Multi-Sensor Information Fusion
  • Affordance Cueing in Perception-Action Framework (Learning)

Framework for Affordance Cueing Framework for Affordance Cueing

Key Contributions Key Contributions

slide-27
SLIDE 27

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

  • Integration of 2D and 3D Affordance Cueing
  • Object Recognition using Bundles of Affordances
  • Learning Hierarchical Representation for Affordance Recognition

Framework for Affordance Cueing Framework for Affordance Cueing

Directions of Future Work Directions of Future Work

slide-28
SLIDE 28

MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008 Computational Perception Group (CAPE)

  • VOCUS Visual Attention (FhG/IAIS)
  • Saliency
  • Triangulation
  • Approach & Pose Behavior
  • Affordances based on Range Images (METU)
  • Features from Range Images
  • Sequencing of Affordances

Framework for Affordance Cueing Framework for Affordance Cueing

Further Work Further Work