SUPERVASION PROJECT SUPERVISION BY OBSERVATION USING INDUCTIVE - - PowerPoint PPT Presentation

▶
supervasion project
SMART_READER_LITE
LIVE PREVIEW

SUPERVASION PROJECT SUPERVISION BY OBSERVATION USING INDUCTIVE - - PowerPoint PPT Presentation

Jun 08, 2023 8 likes •254 views

SUPERVASION PROJECT SUPERVISION BY OBSERVATION USING INDUCTIVE PROGRAMMING DAVID NIEVES CORDONES daniecor@dsic.upv.es CARLOS MONSERRAT ARANDA cmonserr@dsic.upv.es JOS HERNNDEZ ORALLO jorallo@dsic.upv.es DAVID NIEVES CORDONES -

slide-1
SLIDE 1

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

SUPERVASION PROJECT

SUPERVISION BY OBSERVATION USING INDUCTIVE PROGRAMMING DAVID NIEVES CORDONES daniecor@dsic.upv.es

CARLOS MONSERRAT ARANDA cmonserr@dsic.upv.es JOSÉ HERNÁNDEZ ORALLO jorallo@dsic.upv.es

slide-2
SLIDE 2

2

INDEX

INTRODUCTION APPROACH RESULTS QUESTIONS

slide-3
SLIDE 3

INTRODUCTION SUPERVASION Project

3

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

DOMAIN

Minimally Invasive Surgery (MIS)

HOW GOAL Automated assistants that after the observation of how an expert performs a task are able to supervise whether

  • ther humans are performing the task correctly, also by
  • bservation.

The system should be capable of learning from

  • bservations.

The learning process should take from one/few demonstrations. The acquired knowledge must be intelligible to humans.

Automated supervision by observation

slide-4
SLIDE 4

4

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

MIS resources for trainees

High-level description Video exercise

INTRODUCTION Domain

slide-5
SLIDE 5

APPROACH Elements of Event Calculus*

5

  • Events

Starts or ends a fluent (e.g., forceps has just crossed the 5 cm line)

  • Low-level fluents

A direct observable property that is true along a period

  • f time (e.g., forceps is inside the 5 cm zone)
  • High-level fluents
  • Extracted from expert explanation (not directly observable) (e.g., approach

the forceps to the bin with the objects)

  • Set true because other fluents (usually directly observable ones) are true.

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

* Artikis, A., Sergot, M., & Paliouras, G. (2015). An event calculus for event recognition. IEEE Transactions on Knowledge and Data Engineering, 27(4), 895-908.

slide-6
SLIDE 6

APPROACH

Elements of Event Calculus

6

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

“System learns the rules that connect the low-level fluents with the high-level fluents.”

Domain approach

initiatesF(clip_object(i1,o1),T) :- holdsAt(open(i1),T), holdsAt(in5(i1,g1),T)

slide-7
SLIDE 7

APPROACH

Supervision by observation

7

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION

The system learns from example performances and the narrative The system uses the acquired knowledge to supervise the training process of the novel surgeons

Knowledge builder [XHAIL]

Event Builder Narrative [Surgeon explanation] General and Specific Knowledge Acquired knowledge

Event detection [RTEC]

Event Builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration]

slide-8
SLIDE 8

8

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event detection [RTEC]

Event Builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration] Knowledge builder [XHAIL]

Event Builder Narrative [Surgeon explanation] General and Specific Knowledge Acquired knowledge

APPROACH

Knowledge acquisition phase

slide-9
SLIDE 9

9

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event detection [RTEC]

Event Builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration] Knowledge builder [XHAIL]

Narrative [Surgeon explanation] General and Specific Knowledge Acquired knowledge

Event Builder

APPROACH

Knowledge acquisition phase

slide-10
SLIDE 10

APPROACH

Knowledge acquisition phase

10

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Event Builder (EB)

  • Events are extracted from video

recordings of experts performing the training task.

  • The EB detects and states the
  • rder in which events happen,

thus generating the corresponding predicates in EC.

Knowledge Builder [XHAIL]

Event Builder

Narrative [Surgeon explanation] General and Specific Knowledge Acquired knowledge

{E1, TF1} {E2, TF2} {E3, TF3} {E4, TF4} {E5, TF5} ….

happensAt(E1, T1). happensAt(E2, T2). happensAt(E3, T3). happensAt(E4, T4). happensAt(E5, T5). … Predicates in EC Events detected

slide-11
SLIDE 11

11

APPROACH

Knowledge acquisition phase

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

slide-12
SLIDE 12

12

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event detection [RTEC]

Event builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration] Knowledge builder [XHAIL] Narrative [Surgeon explanation]

General and Specific Knowledge Acquired knowledge Event Builder

APPROACH

Knowledge acquisition phase

slide-13
SLIDE 13

13

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Narrative [Surgeon explanation]

  • Extracted

from the expert high-level explanation and translated to a sequence of fluents in EC.

  • Properties that hold or not at

any instant of time

Knowledge Builder [XHAIL]

Event Builder Narrative [Surgeon explanation] General and Specific Knowledge Acquired knowledge

#example not holdsAt(F1, T1). #example not holdsAt(F1, T2). #example holdsAt(F1, T3). #example holdsAt(F1, T4). #example not holdsAt(F1, T5). #example not holdsAt(F1, T6). ….

APPROACH

Knowledge acquisition phase

slide-14
SLIDE 14

14

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event detection [RTEC]

Event Builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration] Knowledge builder [XHAIL]

Acquired knowledge Event Builder Narrative [Surgeon explanation]

General and Specific Knowledge

APPROACH

Knowledge acquisition phase

slide-15
SLIDE 15

Event Builder Narrative [Surgeon explanation] General and Specific Knowledge Acquired knowledge

15

Knowledge Builder [XHAIL]

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

PREDICATES holdsAt(F, T) happensAt(E, T) initiates(E, F, T) terminates(E, F, T) initially(F) stoppedIn(T1, F, T2) GENERAL AXIOMS holdsAt(F, T) :- initially(F), not stoppenIn(0, F, T) holdsAt(F, T) :- happensAt(E, T1), not stoppedIn(T1, F,T),T1<T stoppedIn(T1, F, T2) :- happensAt(E, T), T1<T, T<T2, terminates(E, F, T). PREDICATES holdsAtF(FF, T). initiatesF(FF, T). terminatesF(FF, T). GENERAL AXIOMS holdsAt(FF, T+1) :- not holdsAt(FF, T), initiatesF(FF, T). holdsAt(FF, T+1) :- holdsAt(FF, T), not terminatesF(FF, T).

High-level fluents Low-level fluents

APPROACH

Knowledge acquisition phase

slide-16
SLIDE 16

16

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION

Knowledge builder [XHAIL]

Event Builder Narrative [Surgeon explanation] General and Specific knowledge in EC Acquired knowledge

Event detection [RTEC]

Event Builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration]

APPROACH

Online supervision phase

slide-17
SLIDE 17

17

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION

Knowledge builder [XHAIL]

Event Builder Narrative [Surgeon explanation] General and Specific knowledge in EC Acquired knowledge

Event detection [RTEC]

Event Builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration]

APPROACH

Online supervision phase

slide-18
SLIDE 18

18

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

Detected fluents

  • The
  • utput
  • f

the Online Supervision phase is the detection

  • f when one high-level fluent

starts and ends.

  • The success of the detection

process depends on whether the initiatesF/2 and/or terminatesF/2

  • f the new executions are inside

the corresponding temporal window.

Event detection [RTEC]

Event Builder

Acquired knowledge Specific Knowledge

Detected fluents [surgeon narration]

APPROACH

Online supervision phase

slide-19
SLIDE 19

19

RESULTS

Example exercise

Evaluation

We have analysed the ability of the system to learn and monitor the MIS high-level fluent clipping.

Exercise Starts just before one object o1 (contained in one bin g1) is clamped by an instrument i1. Scenarios

Two performance examples: Scenario A & Scenario B

Finishes just when this object o1 is released (in a second bin g2).

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

slide-20
SLIDE 20

20

RESULTS

Scenarios

Scenario A Scenario B

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

https://github.com/cmonserr/SUPERVASION More examples

slide-21
SLIDE 21

21

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

RESULTS

Rules Scenario A -> Scenario B

initiatesF(clip_object(i1,o1),T) :- holdsAt(open(i1),T), holdsAt(in5(i1,g1),T) terminatesF(clip_object(i1,o1),T) :- holdsAt(waiting(i1),T), holdsAt(in5(i1,g2),T) Rules Scenario A

slide-22
SLIDE 22

22

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

QUESTIONS

Open research questions

How can we automate the synchronisation / matching of the high-level fluents (narratives) from the high-level descriptions (natural language)? Any alternative to this approach? How can we generalise from multiple examples (different performances) of the same exercise? Is it possible to extract constraints between low-level events from a high-level description?

slide-23
SLIDE 23

23

DAVID NIEVES CORDONES - daniecor@dsic.upv.es

How can we identify whether all the low-level fluents are proper for this task? Is IP suitable for this task?

?

QUESTIONS

Open research questions

slide-24
SLIDE 24

24

Thanks for your attention

DAVID NIEVES CORDONES daniecor@dsic.upv.es

CARLOS MONSERRAT ARANDA cmonserr@dsic.upv.es JOSÉ HERNÁNDEZ ORALLO jorallo@dsic.upv.es