Final Project Review Overview Sankt Augustin, February 15th, 2008 - PowerPoint PPT Presentation

Final Project Review Overview Sankt Augustin, February 15th, 2008 Erich Rome, Adaptive Reflective Teams Department ... and all others of the MACS consortium

MACS Project Overview – Reporting Period 3 1. MACS Approach MACS Approach 1. Introduction 2. MACS Approach to Affordance-based Robot Control 2. Work in Period 3 Work in Period 3 1. Work Performed: WP overview, WP1&6 details 3. Conclusion Conclusion 2 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

MACS Project Overview – Reporting Period 3 1. MACS Approach MACS Approach 1. Introduction 2. MACS Approach to Affordance-based Robot Control 2. Work in Period 3 Work in Period 3 1. Work Performed: Achievements, WP overview, WP1&6 details 3. Conclusion Conclusion 3 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.1 Introduction Cognitive abilities:  Using known (and unknown) objects in a variety of ways  Finding alternative solutions for a given task ➜ Ability to improvise; practical aspect of intelligence Source: www.flickr.com  Exploiting specific interaction possibilities that the environment offers humans J.J. Gibson: Affordances 4 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.1 Introduction Research Questions in MACS:  How can we design a “cognitive” mobile robot system with manipulation capabilities that can, e.g.,  find alternative solutions for a given task,  interact with known and unknown objects in a meaningful and goal-directed way, and  uses perception methods that are tailored for its tasks and its action capabilities, i.e. that are grounded in its actions?  Valid approach: Draw inspiration from Cognitive Science ➜ Affordances 5 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.1 Introduction – Gibson’s affordances Characterizing affordances  The “picked-up” visual information includes visible functions (or utilities) of an object / thing / entity)  These functions can be described using abstract features (related to physical properties of the animal)  The same object / thing / entity can offer different functions for different animals 6 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.1 Introduction – Gibson’s affordances Some conclusions:  The characteristic feature set related to a function sit! works like a matched filter across various object categories. sit!  The ability to perceive affordances, i.e., to perceive functions of entities in the world, enables more possibilities for action: sit! sit! An animal (or agent) could even guess what to do Affordance “sitable” offered by various with entities that it has never before perceived. entities in the environment 7 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach Project goals:  Explore how affordances can be used for robot control  Perception, learning, representation and goal-oriented use of affordances  Realisation of a complete affordance-inspired control system  Experimental evaluation  Proof of concept with a simulated and a real robot in a demonstrator scenario 8 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach MACS stance:  Using affordances for deliberative control of a robotic system requires a representation  An explicit representation of affordances benefits from a formalization of the affordance notion.  Affordances are modelled as relations between the abilities of an agent and features in the environment (based on proposals of Stoffregen, 2000 ff , and Chemero, 2003 ff ) 9 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

Sahin et al. (2007): Adaptive Behavior 15(4) . 1.2 MACS Approach – Formalisation agent environment Behavior Entity Outcome Affordance Relation: (Entity, Behavior, Outcome) Entity: Perception aspect of the affordance. Relevant cues of the environment that provide support for the affordance. Behavior: Action aspect of the affordance in a robotic agent sense. Outcome: Outcome (or effect) of the agent’s acting upon an affordance (of applying the behavior) 10 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Formalisation Agent affordance:  An agent must be able to perceive an agent affordance.  “Entity” is usually not necessarily equal to “object”. 11 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

Sahin et al. (2007): Adaptive Behavior 15(4 ) 1.2 MACS Approach – Equivalences Liftability affordance: Blue can Lift Entity Lifted Lift Black can Entity Lifted Entity equivalence: A behavior applied on different entities produces the same outcome. {entity}: liftable entity characteristics, common, e.g., to blue and black can ({entity}, behavior, outcome) 12 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Affordance Representation Definition:  (Agent) Affordance Representation: data structure (cue descriptor, behavior descriptor, outcome descriptor) , where • cue descriptor : entity representation, containing pairs of attributes and associated value ranges, • behavior descriptor : reference to a robot behavior – reactive or high-level –, plus an optional set of behavior parameters, • outcome descriptor : analogous to cue descriptor Rome et al. (2006): D2.2.2 13 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Representation Cue descriptor:  The cue descriptor will contain acquired information that is characteristic for the agent affordance at hand.  Once learned, the cue descriptor can be used to perceive an affordance. ( ➜ (matched) filter)  The agent may refine the cue descriptor later when it makes new experiences. 14 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

R. Breithaupt et al. (FhG/AIS) 1.2 MACS Approach – Robot K URT 3D Robotic agent K URT 3D: Sensors:  2 cameras  3D Laser scanner  Distance transducers  weight sensor  more … Actuators:  6 wheels, 2 drive motors  3-DOF crane with & electromagnetic gripper 15 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Demonstrator Scenario Demonstrator scenario:  Two separated regions (“rooms”)  Sliding door, operated by switch  Switch triggerable by weight (adjustable)  Test objects:  Cylinders, spheres, boxes  Different tops, sizes, weights, color combinations Real MACSim 16 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Basic Skills Basic perception and action capabilities (skills):  Visual attention (bottom-up and top-down) Perception module Feature detectors  Feature detectors (single sensor, multi-sensor) as Computational Units Paletta, Fritz, May, Ugur et al. (JR_DIB, FhG/IAIS, METU-KOVAN)  Pre-processing scanner data (free space, obstacles)  Roaming (uses Drive , Brake , Turn ) Behavior system Basic skills  Approach_pose  Push , Lift , Drop , Carry , Stack , ... C. Lörken et al.  Remote_control (for manual teaching) (FhG/IAIS – UOS) 17 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Agent Affordances Initial set of agent affordances:  traversable (offered by free space)  pass(-through)-able (region between things)  push-able (thing)  lift-able (thing)  place-able (offered by region) New and refined:  switch-trigger-able (lift-able + place-able)  removable-from-switch (lift-able + place-able)  traversable (free space + push-able) 18 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Characterizing Affordances Characterizing some affordances:  Lift-able characterization:  size limited by scanner resolution and FOV  magnetizable  limited weight (< 1kg)  flat top  color, shape may vary in limits of perceptual abilities 19 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

1.2 MACS Approach – Exploration and Application Modes of operation:  Simple pre-coded behaviors (basic skills)  Acquiring affordance knowledge (exploration)  autonomous or  controlled by human  Goal-directed (plan-based) affordance usage (application) Simple pre- Interact Discover Use relations coded with the general in goal-directed behaviors environment relations behavior Exploration Application 20 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

L. Paletta, G. Fritz Joanneum Research 1.2 MACS Approach – Bottom-up Perception Bottom-up perception: Applying feature detectors  Feature detectors (computational units):  Saliencies in camera, range and remission images  Color blobs  SIFT categories  Blob size ratios  Behavior activation Cue Cue SIFT regions Histogram categories  Weight sensor  …  Find characteristic subset 21 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS

Dorffner, Kintzler, Irran OFAI 1.2 MACS Approach – Knowledge Acquisition Acquiring knowledge on affordance relations Bottom-up acquisition Performing (pre-coded) behavior (B)  Perceiving own actions ( ➜ learning)  Aff. Repres. Repository Gaining knowledge about cues (C) ( ➜ learning)  Gaining knowledge about outcomes (O) ( ➜ learning)  Learning module Building C-B-O Affordance Representation Repository  Perception module Feature detectors Behavior system 22 MACS_Y3_Final_Review_Overview.ppt FP6-004381-MACS