SUPERVASION PROJECT SUPERVISION BY OBSERVATION USING INDUCTIVE PROGRAMMING DAVID NIEVES CORDONES daniecor@dsic.upv.es CARLOS MONSERRAT ARANDA cmonserr@dsic.upv.es JOSÉ HERNÁNDEZ ORALLO jorallo@dsic.upv.es DAVID NIEVES CORDONES - daniecor@dsic.upv.es
INTRODUCTION APPROACH INDEX RESULTS QUESTIONS 2
INTRODUCTION SUPERVASION Project Automated assistants that after the observation of how an expert performs a task are able to supervise whether GOAL other humans are performing the task correctly , also by observation. Automated supervision by observation The system should be capable of learning from observations. The learning process should take from one/few HOW demonstrations. The acquired knowledge must be intelligible to humans. Minimally Invasive Surgery (MIS) DOMAIN 3 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
INTRODUCTION Domain High-level description MIS resources for trainees Video exercise 4 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Elements of Event Calculus* - Events Starts or ends a fluent ( e.g., forceps has just crossed the 5 cm line ) - Low-level fluents A direct observable property that is true along a period of time ( e.g., forceps is inside the 5 cm zone ) - High-level fluents - Extracted from expert explanation (not directly observable) ( e.g., approach the forceps to the bin with the objects ) - Set true because other fluents (usually directly observable ones) are true. * Artikis, A., Sergot, M., & Paliouras, G. (2015). An event calculus for event recognition. IEEE Transactions on Knowledge and Data Engineering, 27(4), 895-908. 5 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Elements of Event Calculus “System learns the rules that connect the low-level fluents with the high-level fluents.” Domain approach initiatesF(clip_object(i1,o1),T) :- holdsAt(open(i1),T), holdsAt(in5(i1,g1),T) 6 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Supervision by observation Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event Event Builder Builder Narrative Acquired [Surgeon knowledge Event explanation] Knowledge detection Specific builder Acquired knowledge Knowledge [XHAIL] Detected [RTEC] General and fluents Specific [surgeon Knowledge narration] The system uses the acquired knowledge to The system learns from example supervise the training process of the novel performances and the narrative surgeons 7 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event Event Builder Builder Narrative Acquired [Surgeon knowledge Event explanation] Knowledge detection Specific builder Acquired knowledge Knowledge [XHAIL] Detected [RTEC] General and fluents Specific [surgeon Knowledge narration] 8 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event Event Builder Builder Narrative Acquired [Surgeon knowledge Event explanation] Knowledge detection Specific builder Acquired knowledge Knowledge [XHAIL] Detected [RTEC] General and fluents Specific [surgeon Knowledge narration] 9 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase Events detected Predicates in EC {E 1 , TF 1 } happensAt(E 1 , T 1 ). {E 2 , TF 2 } happensAt(E 2 , T 2 ). {E 3 , TF 3 } happensAt(E 3 , T 3 ). {E 4 , TF 4 } happensAt(E 4 , T 4 ). Event happensAt(E 5 , T 5 ). {E 5 , TF 5 } Builder … … . Event Builder (EB) Narrative [Surgeon - Events are extracted from video Knowledge explanation] recordings of experts performing Builder Acquired the training task. knowledge [XHAIL] General and - The EB detects and states the Specific Knowledge order in which events happen, thus generating the corresponding predicates in EC. 10 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase 11 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event Event Builder builder Narrative Acquired [Surgeon knowledge Event explanation] Knowledge detection Specific builder Acquired knowledge Knowledge [XHAIL] Detected [RTEC] General and fluents Specific [surgeon Knowledge narration] 12 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase Narrative [Surgeon explanation] Event - Extracted from the expert Builder high-level explanation and translated to a sequence of fluents in EC. Narrative [Surgeon Knowledge explanation] - Properties that hold or not at Builder Acquired any instant of time knowledge [XHAIL] General and Specific #example not holdsAt(F 1 , T 1 ). Knowledge #example not holdsAt(F 1 , T 2 ). #example holdsAt(F 1 , T 3 ). #example holdsAt(F 1 , T 4 ). #example not holdsAt(F 1 , T 5 ). #example not holdsAt(F 1 , T 6 ). … . 13 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event Event Builder Builder Narrative Acquired [Surgeon knowledge Event explanation] Knowledge detection Specific builder Acquired knowledge Knowledge [XHAIL] Detected General and [RTEC] fluents Specific [surgeon Knowledge narration] 14 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Knowledge acquisition phase Low-level fluents PREDICATES holdsAt(F, T) happensAt(E, T) Event initiates(E, F, T) terminates(E, F, T) initially(F) stoppedIn(T1, F, T2) Builder GENERAL AXIOMS holdsAt(F, T) :- initially(F), not stoppenIn(0, F, T) holdsAt(F, T) :- happensAt(E, T1), not stoppedIn(T1, F,T),T1<T Narrative stoppedIn(T1, F, T2) :- happensAt(E, T), T1<T, T<T2, [Surgeon terminates(E, F, T). Knowledge explanation] Builder Acquired knowledge High-level fluents [XHAIL] General and PREDICATES Specific holdsAtF(FF, T). Knowledge initiatesF(FF, T). terminatesF(FF, T) . GENERAL AXIOMS holdsAt(FF, T+1) :- not holdsAt(FF, T), initiatesF(FF, T). holdsAt(FF, T+1) :- holdsAt(FF, T), not terminatesF(FF, T). 15 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Online supervision phase Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event Event Builder Builder Narrative Acquired [Surgeon knowledge Event explanation] Knowledge detection Specific builder Acquired knowledge Knowledge [XHAIL] Detected General and [RTEC] fluents Specific knowledge in [surgeon EC narration] 16 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Online supervision phase Phase 1: KNOWLEDGE ACQUISITION Phase 2: ONLINE SUPERVISION Event Event Builder Builder Narrative Acquired [Surgeon knowledge Event explanation] Knowledge detection Specific builder Acquired knowledge Knowledge [XHAIL] Detected General and [RTEC] fluents Specific knowledge in [surgeon EC narration] 17 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
APPROACH Online supervision phase Detected fluents Event - The output of the Online Builder Supervision phase is the detection of when one high-level fluent starts and ends . Acquired knowledge Event - The success of the detection detection Specific process depends on whether the Knowledge initiatesF/2 and/or terminatesF/2 Detected [RTEC] of the new executions are inside fluents [surgeon the corresponding temporal narration] window. 18 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
RESULTS Example exercise We have analysed the ability of the system to learn and monitor Evaluation the MIS high-level fluent clipping . Starts just before one object o1 (contained in one bin g1) is clamped by an instrument i1. Exercise Finishes just when this object o1 is released (in a second bin g2). Scenarios Two performance examples: Scenario A & Scenario B 19 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
RESULTS Scenarios Scenario A Scenario B https://github.com/cmonserr/SUPERVASION More examples 20 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
RESULTS Rules Scenario A -> Scenario B Rules Scenario A initiatesF(clip_object(i1,o1),T) :- holdsAt(open(i1),T), holdsAt(in5(i1,g1),T) terminatesF(clip_object(i1,o1),T) :- holdsAt(waiting(i1),T), holdsAt(in5(i1,g2),T) 21 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
QUESTIONS Open research questions Any alternative to this approach? How can we automate the synchronisation / matching of the high-level fluents (narratives) from the high-level descriptions (natural language)? How can we generalise from multiple examples (different performances) of the same exercise? Is it possible to extract constraints between low-level events from a high-level description? 22 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
QUESTIONS Open research questions How can we identify whether all the low-level fluents are proper for this task? Is IP suitable for this task? ? 23 DAVID NIEVES CORDONES - daniecor@dsic.upv.es
Thanks for your attention DAVID NIEVES CORDONES daniecor@dsic.upv.es CARLOS MONSERRAT ARANDA cmonserr@dsic.upv.es JOSÉ HERNÁNDEZ ORALLO jorallo@dsic.upv.es 24
Recommend
More recommend
