A Deliberation Layer for Instantiating Robot Execution Plans from - PowerPoint PPT Presentation

A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions June 10th, 2013 D. Di Marco ∗ , R.J.M. Janssen ◦ , A.C. Perzylo ⊛ , M.J.G. van de Molengrafu ◦ , P. Levi ∗ ◦ Eindhoven University of Technology, The Netherlands ⊛ Technische Universität München, Germany ∗ Universität Stuttgart, Germany dimarco@ipvs.uni-stuttgart.de

Motivation Primary question : How can task descriptions be shared among different service robot platforms in different environments? • Represent task descriptions in an abstract way ( action recipe ) • Decouple from environment & robot hardware specifics • Store information in a globally accessible database How use these to generate execution plans tailored to the current situation? 2 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

. Example Recipe . . knowrob:occursBeforeInOrdering MoveBaseToGraspPose, Facts: knowrob:PartialOrdering-Strict Types: Individual: ServeADrinkOrder10 knowrob:occursAfterInOrdering GraspBottle roboearth:IntentionalAction SubClassOf: robotPose-handover1) (knowrob:toLocation some EquivalentTo: Class: BringSomethingSomewhere ... Action Recipe Abstract (i.e. independent of robot platform BringSomethingSomewhere and environment) task descriptions 1 PickupSomething • Formulated in OWL , based on a common ontology NavigateToHandover • Can be composed of other recipes or primitive actions, sub-actions are partially ordered HandoverSomething • Annotated with description of requirements, to check whether a RetractArmToParking given robot is capable of executing these 1 M. Tenorth et al.: ”The RoboEarth language: Representing and Exchanging Knowledge about Actions, Objects, and Environments”, ICRA 2012 3 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Types: ... EquivalentTo: knowrob:occursAfterInOrdering GraspBottle knowrob:occursBeforeInOrdering MoveBaseToGraspPose, Facts: robotPose-handover1) SubClassOf: roboearth:IntentionalAction knowrob:PartialOrdering-Strict Class: BringSomethingSomewhere Individual: ServeADrinkOrder10 Action Recipe Abstract (i.e. independent of robot platform BringSomethingSomewhere and environment) task descriptions 1 . PickupSomething Example Recipe • Formulated in OWL , based on a . common ontology NavigateToHandover • Can be composed of other recipes or (knowrob:toLocation some primitive actions, sub-actions are partially ordered HandoverSomething • Annotated with description of requirements, to check whether a RetractArmToParking given robot is capable of executing . these 1 M. Tenorth et al.: ”The RoboEarth language: Representing and Exchanging Knowledge about Actions, Objects, and Environments”, ICRA 2012 3 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Action Instantiation • Task: navigate into the next room • How to infer that ”Navigate” has to be instantiated as ”open door, pass open door”? ⇒ we need more knowledge 4 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Action Recipe Annotation NAVIGATE-TASK ?robot ?goal BringSomethingSomewhere PickupSomething already there? No Operation NavigateToHandover OpenDoor NavInReachOfDoorButton door between? PushDoorButton HandoverSomething NavigateToTarget RetractArmToParking default case MoveToPrimitive HTN decompositions for actions with preconditions & effects, fully encoded in OWL and linked to recipes. 5 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Plan Instantiation Process • Semantic Maps: basic topology of environment Semantic SRDL Robot Recipe HTN Action • SRDL: Semantic Robot Map Description Annotations Recipes Description Language 1 HTN world state HTN Planning description Domain • Off-the-shelf HTN planner: HTN Planner SHOP2 1 • CRAM Plan Language (CPL) to Symbolic Plan specify execution plans CPL Execution Plan 1 L. Kunze et al.: ”Towards semantic robot description languages”. ICRA 2011 2 D. Nau et al.: ”SHOP2: An HTN planning system”. JAIR, 2003 6 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Plan Instantiation Process • Semantic Maps: basic topology OWL of environment Semantic SRDL Robot Recipe HTN Action • SRDL: Semantic Robot Map Description Annotations Recipes Description Language 1 HTN world state HTN Planning description Domain • Off-the-shelf HTN planner: HTN Planner SHOP2 1 • CRAM Plan Language (CPL) to SHOP2 Symbolic Plan specify execution plans CPL Execution CRAM-PL Plan 1 L. Kunze et al.: ”Towards semantic robot description languages”. ICRA 2011 2 D. Nau et al.: ”SHOP2: An HTN planning system”. JAIR, 2003 6 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

CRAM Plan Language 1 Extension of Common Lisp for robot execution plans • Control structures for executing/synchronizing tasks in parallel • Implementation of the Rete algorithm for Prolog-like reasoning facilities • Designators : symbolic description of objects, locations, task parts • Based on ideas from Firby 2 • Created from action specifications in recipe • Get resolved at the latest possible instant during plan execution • Independent of Hardware/Environment 1 M. Beetz, L. Mösenlechner, M. Tenorth, ”CRAM - A Cognitive Robot Abstract Machine for Everyday Manipulation in Human Environments”. IROS, 2010 2 R.J. Firby, ”Adaptive execution in complex dynamic worlds”. PhD thesis, Yale University, 1989 7 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Architecture Cloud Semantic SRDL Action Recipe HTN Storage Maps Robot Descr. Recipes Annotations SeRQL Interface Human Machine Interface Query Response Task Definition Comm Detections Local Inference System Query Planning Domain Simulated Extraction Perception Local Local Know- World Model ledge Base Problem Domain Response Query Response SHOP2 Planner CRAM executive Parametrized SHOP2 Plan Process Modules CPL Plans ROS Actionlib CPL Plan Calls Generation ROS component layer 8 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Simulation Experiment • Gazebo simulator 1 used • Fraunhofer IPA’s Care-o-Bot 3-4 (1 arm, 1 tray) and TU/e’s Amigo (2 arms) prototype • Task description: ”bring something somewhere” • Both robots use same top-level recipe, but need to select different decompositions 1 http://gazebosim.org 9 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Experiment Environment The door can be opened by touching the green button. The used arm has to be free. 10 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Differences in generated Plans NAVIGATE-TASK MoveTo No door: navigation between rooms trivial 11 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Differences in generated Plans NAVIGATE-TASK MoveTo OPEN-DOOR FREE-ARMS-FROM-ENVIRONMENT MoveTo NoOP FREE-ARMS-FROM-ENVIRONMENT FREE-ARM-FOR-GRASPING ChangeGripperState MoveArmTo TuckArm NoOP NoOP 2 manipulators: grasp object with lefu, open door with right arm 11 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Differences in generated Plans NAVIGATE-TASK MoveTo OPEN-DOOR FREE-ARMS-FROM-ENVIRONMENT MoveTo PickupObjectFromTray FREE-ARMS-FROM-ENVIRONMENT FREE-ARM-FOR-GRASPING ChangeGripperState MoveArmTo TuckArm NoOP PutObjectOnTray 1 manipulator + tray: put object on tray, open door, pickup object 11 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Video - Video Demo - 12 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Conclusion • Increase applicability of abstract task descriptions using off-the-shelf HTN planning • Separation of sources of knowledge • Experiment required about 4K lines of OWL • Classic symbolic reasoning, e.g. no temporal aspects Future work: • Apply approach to more difficult scenarios • Integrate plan generation and task execution for more robust behavior 13 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions

Discussion Thank you for your attention 14 / 14 A Deliberation Layer for Instantiating Robot Execution Plans from Abstract Task Descriptions