Answer Set Programming and Human-Robot Interactions: Challenges and - - PowerPoint PPT Presentation

Answer Set Programming and Human-Robot Interactions: Challenges and Opportunities

Tran Cao Son

Computer Science, New Mexico State University, Las Cruces, NM (Joint work with: Chitta Baral, Michael Gelfond, and Arindam Mitra)

Dresden 2016

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 1 / 29

Outline

1

Motivation

2

Answer Set Programming

3

Addressing the Challenges Extension to Agent Architecture NLP to Goal Description Planning with Goal Description What To Do When Planner Fails?

4

Conclusions and Future Work

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 2 / 29

Motivation

Outline

1

Motivation

2

Answer Set Programming

3

Addressing the Challenges Extension to Agent Architecture NLP to Goal Description Planning with Goal Description What To Do When Planner Fails?

4

Conclusions and Future Work

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 3 / 29

Motivation

General Intelligent Agent Architecture

Given a goal

1

Create a plan

2

Execute the first action

3

Observe the environment, if

• bservations are as expected

continue with the plan;

• therwise, back to 1.

Planner

Execution Monitoring

Deliberative agent Goal Success/Fail

Characteristics

Communication are done via logical formulas. No interaction between agents and human during the execution of a goal. Response to a given goal is either success or fail.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 4 / 29

Motivation

General Intelligent Agent Architecture

Given a goal

1

Create a plan

2

Execute the first action

3

Observe the environment, if

• bservations are as expected

continue with the plan;

• therwise, back to 1.

Planner

Execution Monitoring

Deliberative agent Goal Success/Fail

Human robot interaction is important!

Several practical applications: emergency rescue missions (helicopter, robots) fire fighters (robots) self-driving cars, assisted robots assistant bots (software, hardware, etc.) ...

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 4 / 29

Motivation

Human-Robot Interaction

Human Robot

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Motivation

Human-Robot Interaction

Human Robot

✲

Commands Add another stack to the table!

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Motivation

Human-Robot Interaction

Human Robot

✲

Commands Add another stack to the table! Planner

✻

Goal

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Motivation

Human-Robot Interaction

Human Robot

✲

Commands Add another stack to the table! Planner

✻

Goal

❄

Plan

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Motivation

Human-Robot Interaction

Human Robot

✲

Commands Add another stack to the table! Planner

✻

Goal

❄

Plan Execution

❄

Plan

✻

Success

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Motivation

Human-Robot Interaction

Human Robot

✲

Commands Add another stack to the table! Planner

✻

Goal

❄

Plan Execution

❄

Plan

✻

Success

✛

Done

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Motivation

Human-Robot Interaction

Human Robot

✲

Commands Add another blue stack of the same height! Planner

✻

Goal

❄

No Plan

✛

Not Done

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Motivation

Challenges

Human Robot

✲

Commands Add another blue stack of the same height! Planner

✻

Goal

❄

No Plan

✛

Not Done What to do if planner fails? How to communicate in NLP? (NLP commands to goals)

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 5 / 29

Answer Set Programming

Outline

1

Motivation

2

Answer Set Programming

3

Addressing the Challenges Extension to Agent Architecture NLP to Goal Description Planning with Goal Description What To Do When Planner Fails?

4

Conclusions and Future Work

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 6 / 29

Answer Set Programming

Logic Programming Rules and Programs

Rule

r : b1 or . . . or bm ← a1, . . . , an, not an+1, . . . , not an+k ai, bj: atom of a first order language L. not a: a negation-as-failure atom (naf-atom).

Program

A program is a set of rules. Use Herbrand semantics.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 7 / 29

Answer Set Programming

Answer Set Semantics

Let π be a program and X be a set of atoms, by πX we denote the program obtained from ground(π) by

1

Deleting from ground(π) any rule a←a1, . . . , an, not an+1, not an+k for that {an+1, . . . , an+k} ∩ X = ∅, i.e., the body of the rule contains a naf-atom not al and al belongs to X; and

2

Removing all of the naf-atoms from the remaining rules.

Definition

A set of atoms X is called an answer set of a program π if X is the minimal model of the program πX.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 8 / 29

Answer Set Programming

General Methodology

Problem Solutions Logic Program Answer Sets Answer Set Solver Modeling Interpretation

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 9 / 29

Answer Set Programming

General Methodology

Problem Solutions Traditional Approach Logic Program Answer Sets Answer Set Solver Modeling Interpretation

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 9 / 29

Answer Set Programming

Answer Set Planning: Block World Domain

Typical ASP encoding: %Defining the time constants and objects time(0..length). block(a). block(b). block(c). %Defining fluents fluent(on(X,Y)):- block(X), block(Y),X!=Y. fluent(onTable(X)):- block(X). fluent(clear(X)):- block(X). ... %Defining actions action(stack(X,Y)):- block(X), block(Y), X!=Y. action(putdown(X)):- block(X). ... %Action effects & precondition :- not holds(clear(X), T), occ(pickup(X), T). holds(-onTable(X), T+1):- occ(pickup(X), T). holds(holding(X), T+1):- occ(pickup(X), T). .... %Planning holds(on(c,b), 0). holds(onTable(b), 0). ... 1 {occ(A,T) : action(A) } 1 :- time(T). goal(T) :- holds(on(a,b), T), holds(on(b,c), T), holds(onTable(c), T). :- not goal(length).

c b a Start c b a Goal

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 10 / 29

Addressing the Challenges

Outline

1

Motivation

2

Answer Set Programming

3

Addressing the Challenges Extension to Agent Architecture NLP to Goal Description Planning with Goal Description What To Do When Planner Fails?

4

Conclusions and Future Work

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 11 / 29

Addressing the Challenges Extension to Agent Architecture

Idea

Agents need to be able to communicate in natural language; and deal with planning failure by informing the human with what is missing so human can help.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 12 / 29

Addressing the Challenges Extension to Agent Architecture

Idea

Agents need to be able to communicate in natural language; and deal with planning failure by informing the human with what is missing so human can help.

Results NLP Module Planner

Execution Monitoring

Failure Analysis Human Commands

Fail Success

Results Response ASP Analysis

Our Proposal: Intelligent Agent Architecture

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 12 / 29

Addressing the Challenges Extension to Agent Architecture

Preliminary Results

Develop a preliminary system for translating commands to NLP using kparser and NL2KR. Define the notion of a planning failure analysis: why does the planner fail to generate a plan? Implement a system for computing most preferred failure analysis.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 13 / 29

Addressing the Challenges NLP to Goal Description

NLP to Goal Description: kparser and NL2KR

kparser: Semantics parser

rich vocabulary for relations (e.g., event-entity, inter-event relations) relations such as prototype_of and instance_of conceptual classes of nodes based on word sense disambiguation (WordNet senses of the words represented by those nodes) semantic roles of the entities present in the text Named entity tagging and syntactic dependency parsing

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 14 / 29

Addressing the Challenges NLP to Goal Description

NL2KR

Sentence Meaning Add another blue stack

• f the same height.

λx. goal_cond(x, op, add) ∧ goal_cond(x, is, stack) ∧ goal_cond(x, color, blue) ∧ goal_cond(x, height, same) ∧ goal_cond( x, type, another). Take my blocks. λx. add_block( has_robot(x) , has(human, x) ∧ block(x)). How about a red stack of the same height as the blue stack? λx. goal_cond( x, is, stack )∧goal_cond( x, color, red )∧goal_cond(x, height, same )∧goal_cond(x, origin, blue )∧goal_cond( x, type, another). Add another stack

• f

the same height as the tallest stack. λx. goal_cond(x, op, add) ∧ goal_cond(x, is, stack) ∧goal_cond(x, height, tallest) ∧goal_cond(x, type, another).

λ-expressions can be translated into any KR-language.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 15 / 29

Addressing the Challenges NLP to Goal Description

How Can It Be Done?

Dictionary

Word CCG Meaning take S/NP λp.λx. add_block(has_robot(x), p @x) blocks NP λx. block(x)

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 16 / 29

Addressing the Challenges NLP to Goal Description

NL2KR: CCG Parser

take my blocks [S] take [S/NP] my blocks [NP] my [NP/N] blocks [N] Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 17 / 29

Addressing the Challenges NLP to Goal Description

NL2KR: Constructing Lambda Expression

take my blocks [S] take [S/NP] my blocks [NP] my [NP/N] blocks [N] H F G

Constructing λ-expression for each node Bottom up:

◮ from dictionary for nodes with known phrases ◮ inverse lambda for others: H = F @ G Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 17 / 29

Addressing the Challenges NLP to Goal Description

NL2KR: Constructing Lambda Expression

take my blocks [S] #x.add_block(has(robot,x),has(human,x)^block(x)) take [S/NP] #p.#x.add_block(has(robot,x),p^@x) my blocks [NP] #x1.has(human,x1)^block(x1) my [NP/N] #x3.#x1.has(human,x1)^x3@x1 blocks [N] #x.block(x)

“Use my blocks” is then translated to add_block(has_robot(S)) ← has(human, S), block(S)

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 17 / 29

Addressing the Challenges NLP to Goal Description

From NLP Commands to Goals

Command: “Add another blue stack of the same height!” generate a set of goal conditions that describes the goal state, e.g.,

◮ goal_condition(S,is,stack)

:- block(S).

◮ goal_condition(S,type,another) :- block(S). ◮ goal_condition(S,color,blue)

:- block(S).

◮ goal_condition(S,height,same) :- block(S).

This requires the understanding about the domain (e.g., a stack can be identified by its top block, a stack is said to be blue if all of its blocks are blue, etc.). provide rules for goal conditions.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 18 / 29

Addressing the Challenges Planning with Goal Description

ASP for Planning with Goal Description

Answer set planning can deal with goal description by providing rules for checking satisfaction of each goal condition. all goal conditions.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 19 / 29

Addressing the Challenges Planning with Goal Description

ASP for Planning with Goal Description

Answer set planning can deal with goal description by providing rules for checking satisfaction of each goal condition. For example,

◮ S represents a stack:

satisfied(S,is,stack,T) :- block(S), time(T), clear(S,T).

◮ Stack identified by S is blue:

satisfied(S,color,blue,T) :- block(S), time(T), color(S,blue),clear(S,T), #count{U:above(U,S,T), not color(U,blue)}==0.

all goal conditions. For example, not_sat_goal(S,T) :- block(S),goal_condition(X,Y,Z), not satisfied(X,Y,Z,T). sat_goal(S, T) :- not not_sat_goal(T). :- X = #count {S : sat_goal(S, length)}, X ==0.

Challenge

Automatically generating the rules for a given goal!

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 19 / 29

Addressing the Challenges What To Do When Planner Fails?

Planning Failure Analysis

The robot cannot complete the command because its planner fails to generate a plan. What should it do?

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 20 / 29

Addressing the Challenges What To Do When Planner Fails?

Planning Failure Analysis

The robot cannot complete the command because its planner fails to generate a plan. What should it do? It needs a planning failure analysis!

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 20 / 29

Addressing the Challenges What To Do When Planner Fails?

Planning Failure Analysis: Previous Approaches

values of some fluents are unknown: assumption-based planning (McIlraith et al.)

◮ identify a set of fluents whose values can be assumed (e.g.,

color of block 14 might be blue );

◮ generate assumption-based plans whose execution conform with

the values of the assumed fluents along its trajectory ([put_on_table(9), stack(14,9) ]).

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 21 / 29

Addressing the Challenges What To Do When Planner Fails?

Planning Failure Analysis: Previous Approaches

values of some fluents are unknown: assumption-based planning (McIlraith et al.)

◮ identify a set of fluents whose values can be assumed (e.g.,

color of block 14 might be blue );

◮ generate assumption-based plans whose execution conform with

the values of the assumed fluents along its trajectory ([put_on_table(9), stack(14,9) ]).

partially satisfying the goal might be sufficient: partial satisfaction planning (Benton et al.)

◮ assume that each subgoal has some utility; and ◮ identify a set of satisfiable subgoals with maximal aggregated utility

([put_on_table(9)]).

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 21 / 29

Addressing the Challenges What To Do When Planner Fails?

Planning Failure Analysis: Our Approach

Action domain used by the planning system could be incomplete: there are actions that the robot does not want to use if they are not needed, e.g., asking to use the human’s blocks paint some blocks blue

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 22 / 29

Addressing the Challenges What To Do When Planner Fails?

Formalization I

Planning failure analysis

A planning problem P = (F, A, I, G) (Fluents, Actions, Initial_State, Goal) needs a planning failure analysis if it has no solution.

Extension to a planning problem

A pair (Afs, Acts) where Afs is a set of fluents and Acts is a set of actions such that F ∩ Afs=∅ and A ∩ Acts = ∅. (∅, {ask_permission}) (Could use any block belonging to human!) (∅, {ask_permission(B) | B is a block}) (Use specific block B) ({paint_available, color(b10, blue)}, {buy_paint, paint_blue(b10)}) (Change the color of the block) ({color(b10, blue)}, ∅) (Wish that the color change)

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 23 / 29

Addressing the Challenges What To Do When Planner Fails?

Formalization II

What is an analysis?

Idea: It should be a part of a possible extension (fluents and action descriptions) It should help to generate plans It should be reasonable

Fluents with Changes

α = [a1, . . . , an] is a solution of a planning problem and a fluent f is changes its value during α execution if its value changes from true to false (false to true) over the trajectory. . . . ¬g f ¬f . . . g

a0 ai aj

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 24 / 29

Addressing the Challenges What To Do When Planner Fails?

Formalization III

Analysis

Given P = F, A, I, G and extension E = (Afs, Acts). (Af, Ac) such that Af ⊆ Afs and Ac ⊆ Acts is called a plan failure analysis for P w.r.t. E ) if there exists an interpretation Af ∗ of the set of fluents Af and P∗ = F ∪ Af, A ∪ Ac, I ∪ Af ∗, G has a solution α such that every f ∈ Af changes its value during α.

Preferred Analysis

(Af, Ac) is a more preferred failure analysis than (Af ′, Ac′), denoted (Af, Ac) ≺ (Af ′, Ac′), if Ac Ac′.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 25 / 29

Addressing the Challenges What To Do When Planner Fails?

Computing (Preferred) Analyses

P = (F, A, I, G) encodes as usual with declarations of actions, fluents, etc. and action generation rule: 1 {occ(A, T): action(A)} 1 :- time(T). Add description of (AF, AC) with the choice rule {is_ac(a)}. for each a ∈ AC Add the rule action(A) :- is_ac(A). Add the following rules for f ∈ AF

{is_fl(f)}. 1 {h(f,0), h(¬ f,0)} 1:- is_fl(f). changed(f) :- h(f,T), h(¬ f,T+1). changed(f) :- h(¬ f,T), h(f,T+1). :- is_fl(f), not changed(f).

Minimizing the set of additional actions number_actions(N) :- #count {A : is_ac(A)}. #minimize {N : number_actions(N)}.

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 26 / 29

Conclusions and Future Work

Outline

1

Motivation

2

Answer Set Programming

3

Addressing the Challenges Extension to Agent Architecture NLP to Goal Description Planning with Goal Description What To Do When Planner Fails?

4

Conclusions and Future Work

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 27 / 29

Conclusions and Future Work

Conclusions

Discuss different challenges in computer human communication Propose a formalization of planning failure analysis. Use answer set programming for computing preferred analyses

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 28 / 29

Conclusions and Future Work

Future Work

From NLP commands to goal-satisfaction ASP rules Generating communications in NLP

Possible Ideas

Utilize NLP tools and develop ASP programs to convert solutions to NLP Create libraries for communication (domain-dependent) Use a more controlled language

Tran Cao Son (NMSU) LP & NLP & Planning Dresden 2016 29 / 29