NECTAR: Knowledge-based Collaborative Active Learning for Activity - PowerPoint PPT Presentation

NECTAR: Knowledge-based Collaborative Active Learning for Activity Recognition Gabriele Civitarese Claudio Bettini Univ. of Milano Univ. of Milano Italy Italy Timo Sztyler Daniele Riboni Heiner Stuckenschmidt Univ. of Mannheim Univ. of Cagliari Univ. of Mannheim Germany Italy Germany 1 PerCom ‘18, Athens, Greece, March 21, 2018

MOTIVATION 2 PerCom ‘18, Athens, Greece, March 21, 2018

IEEE International Conference on Pervasive Computing and Communications Scenario 2016 Recognizing activities of daily living in a smart-home to support healthcare, home automation, a more independent life, … We rely on unobtrusive sensors … 3 PerCom ‘18, Athens, Greece, March 21, 2018

State of the Art and Open Issues Most activity recognition systems rely on … … supervised-based approaches: acquire expensive labeled data sets often user/environment-specific … knowledge-based approaches: require a significant effort in knowledge engineering not flexible questionable if such models could cover different environments and modes of execution 4 PerCom ‘18, Athens, Greece, March 21, 2018

Our solution: NECTAR kNowledge-basEd Collaborative acTive learning for Activity Recognition It overcomes drawbacks of supervised-based approach not user/environment-specific, no expensive data set, … It relies on semantic correlations probabilistic dependencies (activities ↔ events) derived from a possibly incomplete ontology It exploits collaborative active learning ...to refine rough correlations inferred by the ontology 5 PerCom ‘18, Athens, Greece, March 21, 2018

MODEL AND SYSTEM 6 PerCom ‘18, Athens, Greece, March 21, 2018

NECTAR’s architecture 3. Collaborative Feedback Aggregation d Feedback item e Home z i l a e n t o a s d r p e u P 1. Probabilistic and 2. Query decision Ontological (entropy-based) Activity Recognition Continuous stream of Sensor Events 7 PerCom ‘18, Athens, Greece, March 21, 2018

1. Probabilistic/ontological activity recognition We rely on ontological reasoning to pre-compute in an offline phase semantic correlations they define probabilistic dependencies between home infrastructure and sensor events A MLN combines those semantic correlations and sensor events to infer the most likely executed activities Continuous Stream of sensor events Sensor Events detected Probabilistic activities Inference Engine semantic correlations (MLN) Offline Semantic Correlation Reasoner (Ontology) 8 PerCom ‘18, Athens, Greece, March 21, 2018

Semantic Correlation Reasoner Ontology / Axioms OWL2 Reasoner infers {turn on stove} is a predictive sensor event type for {Prepare hot meal} and {Prepare tea} interact SC Matrix stove silverware_drawer freezer Hot meal 0.5 0.33 0.5 prepare Cold meal 0.0 0.33 0.5 Tea 0.5 0.33 0.0 9 PerCom ‘18, Athens, Greece, March 21, 2018

Issues of this approach Semantic correlations are computed based on an ontology written by knowledge engineers (humans) it is very likely that the ontology is incomplete it is hence questionable if it can cover different environments/mode of execution Our goal is to refine and improve semantic correlations thanks to collaborative active learning! 10 PerCom ‘18, Athens, Greece, March 21, 2018

2. Query decision Collaborative Feedback Labeled segment Aggregation Personalized update Query Query decision Semantic correlations ... (entropy-based) Feedback Segment Online rule-based segmentation Sensor events Continuous Stream of Sensor Events 11 PerCom ‘18, Athens, Greece, March 21, 2018

Online rule-based segmentation We continuously segment the stream of sensor events based on knowledge-base conditions (e.g., interaction with objects, time gaps, changes of room) those conditions aim to generate segments which cover at most one activity instance 12 PerCom ‘18, Athens, Greece, March 21, 2018

Query decision For each segment we derive a probability distribution over activities by mining semantic correlations segments with high entropy values are queried to the inhabitant When H(S) is over a certain threshold we ask to the inhabitant the actual label of the segment S 13 PerCom ‘18, Athens, Greece, March 21, 2018

3. Collaborative Feedback Aggregation Labeled segments are transmitted to a cloud service by the participating homes it stores feedback items : correspondence between sensor event types and activities Periodically, a personalized update is transmitted to each home it contains reliable feedback items provided by similar environments 14 PerCom ‘18, Athens, Greece, March 21, 2018

Personalized update To include only reliable feedback items in an update, we consider only whose support is larger than a threshold support is a value which indicates how many times the feedback was provided from different similar homes We associate to each feedback item in an update: its predictiveness: computed as the normalization of support values its estimated similarity: the median value of similarity between origin/target environments 15 PerCom ‘18, Athens, Greece, March 21, 2018

Semantic Correlation Updater Each home receives periodically a set of personalized feedback items predictiveness is used to provide a semantic correlation to those event types for which the original ontology did not provide a starting correlation estimated similarity is used to scale semantic correlations of an event type which were originally computed by the ontology 16 PerCom ‘18, Athens, Greece, March 21, 2018

EXPERIMENTS 17 PerCom ‘18, Athens, Greece, March 21, 2018

IEEE International Conference on Pervasive Computing and Communications Data Set 2016 We consider a well-known data set … CASAS • Interleaved ADLs of twenty-one subjects • Sensors: movement, water, interaction, door, phone • Activities: fill medications dispenser, watch DVD, water plants, answer the phone, clean, choose outfit, … We apply leave-one-subject-out cross validation: in each fold we collect feedback from 20 subjects to update semantic correlations for the remaining one 18 PerCom ‘18, Athens, Greece, March 21, 2018

Recognition results (F1 score) 19 PerCom ‘18, Athens, Greece, March 21, 2018

Improvement of collaborative active learning 20 PerCom ‘18, Athens, Greece, March 21, 2018

Entropy threshold VS number of queries 21 PerCom ‘18, Athens, Greece, March 21, 2018

DISCUSSION / FUTURE WORK 22 PerCom ‘18, Athens, Greece, March 21, 2018

IEEE International Conference on Pervasive Computing and Communications Discussion 2016 Results with a well-known dataset were positive, but... • … contextual aspects should be taken in account to evaluate whether to ask a feedback e.g., number of queries already been asked, current mood, availability • ...user interfaces need to be designed e.g., vocal interfaces • … knowledge engineering is still required (build starting ontology) existing smart-home ontologies can be reused 23 PerCom ‘18, Athens, Greece, March 21, 2018

Future Work Data outsourced to the cloud service is sensitive … … we will investigate solutions based on homomorphic encryption or secure multi-party computation We also aim to extend our system … … learning semantic correlations also for temporal patterns 24 PerCom ‘18, Athens, Greece, March 21, 2018

THANKS FOR YOUR ATTENTION! 25 PerCom ‘18, Athens, Greece, March 21, 2018

BACKUP SLIDES 26 PerCom ‘18, Athens, Greece, March 21, 2018

Entropy threshold VS F1 score 27 PerCom ‘18, Athens, Greece, March 21, 2018

Feedback support threshold vs F1 score 28 PerCom ‘18, Athens, Greece, March 21, 2018

14.09.2016 MLN / MAP Inference Observed predicates → 0.5: hot meal → 0.5: cold meal ADL → 0.0: tea → 0.5: hot meal hot meal? Event 1: opens freezer (1:00pm) → 0.0: cold meal Event 2: turns on stove (1:02pm) cold meal? → 0.5: tea tea? Hidden predicates belong to ADL & Sensor Event Sensor Event Hot meal Freezer Stove 29 PerCom ‘18, Athens, Greece, March 21, 2018

Closet I 01 I 02 I 04 I 06 Living / Dining Room D 07 I 07 M 12 M 13 M 14 AD1 B AD1 C D 11 M 15 M 16 M 17 M 18 M 51 Kitchen P 01 M 11 M 10 M 09 D 08 D 10 D 09 I 08 I 08 I 09 M 08 M 19 Storage Closet D 12 M 05 M 06 M 07 M 01 M 23 M 22 M 21 M 24 I 05 M 04 M 03 M 02 I 03

14.09.2016 MLN Model (detailed) Ontological constraints time-aware inference PPM Matrix temporal *SemanticCorrelation knowledge-based Statistical analysis of events *InstanceCandidate / *Event Observed predicates *Event (SenEvent, EventType, Time) *SemanticCorrelation (SenEvent, ADL, ActivClass, p) *InstanceCandidate (ADL, Start, Stop) Hidden predicates OccursIn (SenEvent, ADL) InstanceClass (ActivClass, ADL) 31 PerCom ‘18, Athens, Greece, March 21, 2018

14.09.2016 Semantic Integration Layer • collects events data from a sensor network • applies preprocessing rules to detect operations Example fridge door sensor signaled “1” → the operation is “opening the fridge” <Event(se 1 , et 1 , t 1 ), … , Event(se k ,et k ,t k )> 32 PerCom ‘18, Athens, Greece, March 21, 2018