New Developments for Robert Assisting Novice Users Even Better in - - PowerPoint PPT Presentation

▶
new developments for robert
SMART_READER_LITE
LIVE PREVIEW

New Developments for Robert Assisting Novice Users Even Better in - - PowerPoint PPT Presentation

Dec 10, 2022 448 likes •1.01k views

New Developments for Robert Assisting Novice Users Even Better in DIY Projects Gregor Behnke , Pascal Bercher, Matthias Kraus, Marvin Schiller, Kristof Mickeleit, Timo H age, Michael Dorna, Michael Dambier, Dietrich Manstetten, Wolfgang

slide-1
SLIDE 1

New Developments for Robert

Assisting Novice Users Even Better in DIY Projects

Gregor Behnke, Pascal Bercher, Matthias Kraus, Marvin Schiller, Kristof Mickeleit, Timo H¨ age, Michael Dorna, Michael Dambier, Dietrich Manstetten, Wolfgang Minker, Birte Glimm, Susanne Biundo

Ulm University / University of Freiburg 30/10/2020

slide-2
SLIDE 2

New Developments for Robert Gregor Behnke, Uni Freiburg

Do-It-Yourself Projects

2/11

slide-3
SLIDE 3

New Developments for Robert Gregor Behnke, Uni Freiburg

Do-It-Yourself Projects

2/11

slide-4
SLIDE 4

New Developments for Robert Gregor Behnke, Uni Freiburg

Do-It-Yourself Projects

2/11

slide-5
SLIDE 5

New Developments for Robert Gregor Behnke, Uni Freiburg

Do-It-Yourself Projects

2/11

slide-6
SLIDE 6

New Developments for Robert Gregor Behnke, Uni Freiburg

Do-It-Yourself Projects

2/11

slide-7
SLIDE 7

New Developments for Robert Gregor Behnke, Uni Freiburg

Do-It-Yourself Projects

2/11

slide-8
SLIDE 8

New Developments for Robert Gregor Behnke, Uni Freiburg

Do-It-Yourself Projects

2/11

slide-9
SLIDE 9

New Developments for Robert Gregor Behnke, Uni Freiburg

Assisting DIYers

3/11

slide-10
SLIDE 10

New Developments for Robert Gregor Behnke, Uni Freiburg

Assisting DIYers

3/11

slide-11
SLIDE 11

New Developments for Robert Gregor Behnke, Uni Freiburg

Assisting DIYers

3/11

slide-12
SLIDE 12

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

4/11

slide-13
SLIDE 13

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

5/11

slide-14
SLIDE 14

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

  • Text

5/11

slide-15
SLIDE 15

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

  • Text
  • Images

5/11

slide-16
SLIDE 16

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

  • Text
  • Images
  • Videos

5/11

slide-17
SLIDE 17

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

  • Text
  • Images
  • Videos
  • Interaction via speech and touch

5/11

slide-18
SLIDE 18

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

  • Text
  • Images
  • Videos
  • Interaction via speech and touch
  • HTN domain provides useful abstraction

5/11

slide-19
SLIDE 19

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

  • Text
  • Images
  • Videos
  • Interaction via speech and touch
  • HTN domain provides useful abstraction
  • Allows changes to the presented plan

5/11

slide-20
SLIDE 20

New Developments for Robert Gregor Behnke, Uni Freiburg

Assistance delivered by Robert

  • Actions are

presented as

  • Text
  • Images
  • Videos
  • Interaction via speech and touch
  • HTN domain provides useful abstraction
  • Allows changes to the presented plan
  • Proactivity using Connected Tool

5/11

slide-21
SLIDE 21

New Developments for Robert Gregor Behnke, Uni Freiburg

Robert’s components

  • (HTN-)Planner

6/11

slide-22
SLIDE 22

New Developments for Robert Gregor Behnke, Uni Freiburg

Robert’s components

  • (HTN-)Planner
  • Ontology Manager

6/11

slide-23
SLIDE 23

New Developments for Robert Gregor Behnke, Uni Freiburg

Robert’s components

  • (HTN-)Planner
  • Ontology Manager
  • Dialogue Manager

6/11

slide-24
SLIDE 24

New Developments for Robert Gregor Behnke, Uni Freiburg

Robert’s components

  • (HTN-)Planner
  • Ontology Manager
  • Dialogue Manager
  • User Interface

6/11

slide-25
SLIDE 25

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

The user might not like the presented plan!

7/11

slide-26
SLIDE 26

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

The user might not like the presented plan! vs

7/11

slide-27
SLIDE 27

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

The user might not like the presented plan! vs

  • We can’t know the user’s preference in beforehand.

7/11

slide-28
SLIDE 28

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

The user might not like the presented plan! vs

  • We can’t know the user’s preference in beforehand.

⇒ React to the user’s whishes.

7/11

slide-29
SLIDE 29

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

8/11

slide-30
SLIDE 30

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user U: I want to use the manual saw instead of the electric one.

8/11

slide-31
SLIDE 31

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

  • Transform request into LTL

U: I want to use the manual saw instead of the electric one.

8/11

slide-32
SLIDE 32

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

  • Transform request into LTL

U: I want to use the manual saw instead of the electric one. φ = G¬sawElectric ∧E sawManually

8/11

slide-33
SLIDE 33

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

  • Transform request into LTL
  • Find plan that satisfies

formula U: I want to use the manual saw instead of the electric one. φ = G¬sawElectric ∧E sawManually

8/11

slide-34
SLIDE 34

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

  • Transform request into LTL
  • Find plan that satisfies

formula U: I want to use the manual saw instead of the electric one. φ = G¬sawElectric ∧E sawManually π | = φ

8/11

slide-35
SLIDE 35

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

  • Transform request into LTL
  • Find plan that satisfies

formula

  • Accumulate set of formulae

for multiple requests U: I want to use the manual saw instead of the electric one. φ = G¬sawElectric ∧E sawManually π | = φ

8/11

slide-36
SLIDE 36

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

  • Transform request into LTL
  • Find plan that satisfies

formula

  • Accumulate set of formulae

for multiple requests U: I want to use the manual saw instead of the electric one. φ = G¬sawElectric ∧E sawManually π | = φ φ1, φ2, . . .

8/11

slide-37
SLIDE 37

New Developments for Robert Gregor Behnke, Uni Freiburg

Changing the Plan

  • Receive request from the

user

  • Transform request into LTL
  • Find plan that satisfies

formula

  • Accumulate set of formulae

for multiple requests U: I want to use the manual saw instead of the electric one. φ = G¬sawElectric ∧E sawManually π | = φ φ1, φ2, . . . What about inconsistencies? ⇒ If ¬∃π : π | = φi, drop the oldest one

8/11

slide-38
SLIDE 38

New Developments for Robert Gregor Behnke, Uni Freiburg

Connected Tool

9/11

slide-39
SLIDE 39

New Developments for Robert Gregor Behnke, Uni Freiburg

Connected Tool

  • Sensors to PSR18Li
  • Gyroscopic
  • Accelerometric
  • Compass

9/11

slide-40
SLIDE 40

New Developments for Robert Gregor Behnke, Uni Freiburg

Connected Tool

  • Sensors to PSR18Li
  • Gyroscopic
  • Accelerometric
  • Compass
  • Trained Neural Network

using data from 12 subjects

9/11

slide-41
SLIDE 41

New Developments for Robert Gregor Behnke, Uni Freiburg

Connected Tool

  • Sensors to PSR18Li
  • Gyroscopic
  • Accelerometric
  • Compass
  • Trained Neural Network

using data from 12 subjects

  • Recognise Activities
  • off
  • screwing
  • drilling
  • drill change
  • battery change
  • other

9/11

slide-42
SLIDE 42

New Developments for Robert Gregor Behnke, Uni Freiburg

Proactivity

We now know what the user is doing!

10/11

slide-43
SLIDE 43

New Developments for Robert Gregor Behnke, Uni Freiburg

Proactivity

We now know what the user is doing! ⇒ Help the user complete his project with suitable dialogue strategies

10/11

slide-44
SLIDE 44

New Developments for Robert Gregor Behnke, Uni Freiburg

Proactivity

We now know what the user is doing! ⇒ Help the user complete his project with suitable dialogue strategies

  • Inform the user of the connected

tool R: You seem to be working with the connected tool for the first time. Don’t worry, I’ll guide you through the steps!

10/11

slide-45
SLIDE 45

New Developments for Robert Gregor Behnke, Uni Freiburg

Proactivity

We now know what the user is doing! ⇒ Help the user complete his project with suitable dialogue strategies

  • Inform the user of the connected

tool

  • Supportive questions

R: I noticed that you were

  • drilling. Did that work?

U: No, it didn’t. R: Ok, do you need additional help? U: Yes, please. R: A video of this project step could help. I’m going to play it for you. (Then a video is played.)

10/11

slide-46
SLIDE 46

New Developments for Robert Gregor Behnke, Uni Freiburg

Proactivity

We now know what the user is doing! ⇒ Help the user complete his project with suitable dialogue strategies

  • Inform the user of the connected

tool

  • Supportive questions
  • Handle inactivity

R: I haven’t seen any tool activity by you in three

  • minutes. Do you need help?

U: ...

10/11

slide-47
SLIDE 47

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

11/11

slide-48
SLIDE 48

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

  • A/B test for proactivity (n = 32, even split)

11/11

slide-49
SLIDE 49

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

  • A/B test for proactivity (n = 32, even split)
  • Subjects were asked to constrcut a wooden key rack using Robert

11/11

slide-50
SLIDE 50

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

  • A/B test for proactivity (n = 32, even split)
  • Subjects were asked to constrcut a wooden key rack using Robert
  • Prior questionnaire, determining inclination for electric saw

11/11

slide-51
SLIDE 51

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

  • A/B test for proactivity (n = 32, even split)
  • Subjects were asked to constrcut a wooden key rack using Robert
  • Prior questionnaire, determining inclination for electric saw
  • Robert did the opposite to elicit requests to change the tool

11/11

slide-52
SLIDE 52

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

  • A/B test for proactivity (n = 32, even split)
  • Subjects were asked to constrcut a wooden key rack using Robert
  • Prior questionnaire, determining inclination for electric saw
  • Robert did the opposite to elicit requests to change the tool

Results:

  • Overall verdict better with proactivity: 3.38/5 vs 3.14/5, but not

significant

11/11

slide-53
SLIDE 53

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

  • A/B test for proactivity (n = 32, even split)
  • Subjects were asked to constrcut a wooden key rack using Robert
  • Prior questionnaire, determining inclination for electric saw
  • Robert did the opposite to elicit requests to change the tool

Results:

  • Overall verdict better with proactivity: 3.38/5 vs 3.14/5, but not

significant

  • 19 subjects used the change feature

11/11

slide-54
SLIDE 54

New Developments for Robert Gregor Behnke, Uni Freiburg

Eval

We evaluated changing the plan and proactivity

  • A/B test for proactivity (n = 32, even split)
  • Subjects were asked to constrcut a wooden key rack using Robert
  • Prior questionnaire, determining inclination for electric saw
  • Robert did the opposite to elicit requests to change the tool

Results:

  • Overall verdict better with proactivity: 3.38/5 vs 3.14/5, but not

significant

  • 19 subjects used the change feature
  • Change was performend according to expectations: 4.29/5

11/11