NECTAR: Knowledge-based Collaborative Active Learning for Activity - - PowerPoint PPT Presentation

▶
nectar knowledge based collaborative active learning for
SMART_READER_LITE
LIVE PREVIEW

NECTAR: Knowledge-based Collaborative Active Learning for Activity - - PowerPoint PPT Presentation

Dec 15, 2022 19 likes •350 views

NECTAR: Knowledge-based Collaborative Active Learning for Activity Recognition Gabriele Civitarese Claudio Bettini Univ. of Milano Univ. of Milano Italy Italy Timo Sztyler Daniele Riboni Heiner Stuckenschmidt Univ. of Mannheim Univ. of

slide-1
SLIDE 1

PerCom ‘18, Athens, Greece, March 21, 2018

NECTAR: Knowledge-based Collaborative Active Learning for Activity Recognition

1

Daniele Riboni

  • Univ. of Cagliari

Italy

Timo Sztyler

  • Univ. of Mannheim

Germany

Gabriele Civitarese

  • Univ. of Milano

Italy

Heiner Stuckenschmidt

  • Univ. of Mannheim

Germany

Claudio Bettini

  • Univ. of Milano

Italy

slide-2
SLIDE 2

PerCom ‘18, Athens, Greece, March 21, 2018

MOTIVATION

2

slide-3
SLIDE 3

PerCom ‘18, Athens, Greece, March 21, 2018 3

IEEE International Conference on Pervasive Computing and Communications 2016

Scenario

Recognizing activities of daily living in a smart-home to support healthcare, home automation, a more independent life, … We rely on unobtrusive sensors …

slide-4
SLIDE 4

PerCom ‘18, Athens, Greece, March 21, 2018 4

State of the Art and Open Issues

Most activity recognition systems rely on … acquire expensive labeled data sets

  • ften user/environment-specific

… supervised-based approaches: require a significant effort in knowledge engineering … knowledge-based approaches: questionable if such models could cover different environments and modes of execution not flexible

slide-5
SLIDE 5

PerCom ‘18, Athens, Greece, March 21, 2018

Our solution: NECTAR

kNowledge-basEd Collaborative acTive learning for Activity Recognition

5

It overcomes drawbacks of supervised-based approach It relies on semantic correlations It exploits collaborative active learning

derived from a possibly incomplete ontology ...to refine rough correlations inferred by the

  • ntology

not user/environment-specific, no expensive data set, … probabilistic dependencies (activities↔ events)

slide-6
SLIDE 6

PerCom ‘18, Athens, Greece, March 21, 2018

MODEL AND SYSTEM

6

slide-7
SLIDE 7

PerCom ‘18, Athens, Greece, March 21, 2018 7

NECTAR’s architecture

  • 3. Collaborative

Feedback Aggregation

Feedback item P e r s

  • n

a l i z e d u p d a t e

Home

Continuous stream of Sensor Events

  • 1. Probabilistic and

Ontological Activity Recognition

  • 2. Query decision

(entropy-based)

slide-8
SLIDE 8

PerCom ‘18, Athens, Greece, March 21, 2018 8

  • 1. Probabilistic/ontological activity recognition

We rely on ontological reasoning to pre-compute in an

  • ffline phase semantic correlations

they define probabilistic dependencies between home infrastructure and sensor events

Offline Semantic Correlation Reasoner (Ontology) Continuous Stream of Sensor Events

A MLN combines those semantic correlations and sensor events to infer the most likely executed activities

Probabilistic Inference Engine (MLN)

sensor events semantic correlations detected activities

slide-9
SLIDE 9

PerCom ‘18, Athens, Greece, March 21, 2018 9

Semantic Correlation Reasoner

stove silverware_drawer freezer Hot meal 0.5 0.33 0.5 Cold meal 0.0 0.33 0.5 Tea 0.5 0.33 0.0

prepare interact Ontology / Axioms {turn on stove} is a predictive sensor event type for {Prepare hot meal} and {Prepare tea} OWL2 Reasoner infers

SC Matrix

slide-10
SLIDE 10

PerCom ‘18, Athens, Greece, March 21, 2018 10

Issues of this approach

Semantic correlations are computed based on an

  • ntology written by knowledge engineers (humans)

it is very likely that the ontology is incomplete it is hence questionable if it can cover different environments/mode of execution

Our goal is to refine and improve semantic correlations thanks to collaborative active learning!

slide-11
SLIDE 11

PerCom ‘18, Athens, Greece, March 21, 2018 11

  • 2. Query decision

Continuous Stream of Sensor Events Online rule-based segmentation Query decision (entropy-based)

Semantic correlations Segment Sensor events Query Feedback

Collaborative Feedback Aggregation

Labeled segment

...

Personalized update

slide-12
SLIDE 12

PerCom ‘18, Athens, Greece, March 21, 2018 12

Online rule-based segmentation

We continuously segment the stream of sensor events based on knowledge-base conditions (e.g., interaction with objects, time gaps, changes of room) those conditions aim to generate segments which cover at most one activity instance

slide-13
SLIDE 13

PerCom ‘18, Athens, Greece, March 21, 2018 13

For each segment we derive a probability distribution

  • ver activities by mining semantic correlations

segments with high entropy values are queried to the inhabitant

Query decision

When H(S) is over a certain threshold we ask to the inhabitant the actual label of the segment S

slide-14
SLIDE 14

PerCom ‘18, Athens, Greece, March 21, 2018 14

  • 3. Collaborative Feedback Aggregation

Labeled segments are transmitted to a cloud service by the participating homes it stores feedback items: correspondence between sensor event types and activities Periodically, a personalized update is transmitted to each home it contains reliable feedback items provided by similar environments

slide-15
SLIDE 15

PerCom ‘18, Athens, Greece, March 21, 2018 15

Personalized update

To include only reliable feedback items in an update, we consider only whose support is larger than a threshold support is a value which indicates how many times the feedback was provided from different similar homes We associate to each feedback item in an update: its predictiveness: computed as the normalization of support values its estimated similarity: the median value

  • f similarity between origin/target

environments

slide-16
SLIDE 16

PerCom ‘18, Athens, Greece, March 21, 2018 16

Semantic Correlation Updater

Each home receives periodically a set of personalized feedback items predictiveness is used to provide a semantic correlation to those event types for which the

  • riginal ontology did not provide a starting

correlation estimated similarity is used to scale semantic correlations of an event type which were

  • riginally computed by the ontology
slide-17
SLIDE 17

PerCom ‘18, Athens, Greece, March 21, 2018

EXPERIMENTS

17

slide-18
SLIDE 18

PerCom ‘18, Athens, Greece, March 21, 2018 18

IEEE International Conference on Pervasive Computing and Communications 2016

Data Set

We consider a well-known data set … CASAS

  • Interleaved ADLs of twenty-one subjects
  • Sensors: movement, water, interaction, door, phone
  • Activities: fill medications dispenser, watch DVD, water plants,

answer the phone, clean, choose outfit, …

We apply leave-one-subject-out cross validation: in each fold we collect feedback from 20 subjects to update semantic correlations for the remaining one

slide-19
SLIDE 19

PerCom ‘18, Athens, Greece, March 21, 2018 19

Recognition results (F1 score)

slide-20
SLIDE 20

PerCom ‘18, Athens, Greece, March 21, 2018 20

Improvement of collaborative active learning

slide-21
SLIDE 21

PerCom ‘18, Athens, Greece, March 21, 2018 21

Entropy threshold VS number of queries

slide-22
SLIDE 22

PerCom ‘18, Athens, Greece, March 21, 2018

DISCUSSION / FUTURE WORK

22

slide-23
SLIDE 23

PerCom ‘18, Athens, Greece, March 21, 2018 23

IEEE International Conference on Pervasive Computing and Communications 2016

Discussion

Results with a well-known dataset were positive, but...

  • … knowledge engineering is still required (build starting
  • ntology)

existing smart-home ontologies can be reused

  • … contextual aspects should be taken in account to

evaluate whether to ask a feedback e.g., number of queries already been asked, current mood, availability

  • ...user interfaces need to be designed

e.g., vocal interfaces

slide-24
SLIDE 24

PerCom ‘18, Athens, Greece, March 21, 2018 24

Future Work

We also aim to extend our system … … learning semantic correlations also for temporal patterns Data outsourced to the cloud service is sensitive … … we will investigate solutions based on homomorphic encryption or secure multi-party computation

slide-25
SLIDE 25

PerCom ‘18, Athens, Greece, March 21, 2018

THANKS FOR YOUR ATTENTION!

25

slide-26
SLIDE 26

PerCom ‘18, Athens, Greece, March 21, 2018

BACKUP SLIDES

26

slide-27
SLIDE 27

PerCom ‘18, Athens, Greece, March 21, 2018 27

Entropy threshold VS F1 score

slide-28
SLIDE 28

PerCom ‘18, Athens, Greece, March 21, 2018 28

Feedback support threshold vs F1 score

slide-29
SLIDE 29

PerCom ‘18, Athens, Greece, March 21, 2018

14.09.2016

29

MLN / MAP Inference

Hidden predicates Observed predicates

Event 1: opens freezer (1:00pm) Event 2: turns on stove (1:02pm)

hot meal? cold meal? tea?

ADL

Sensor Event Stove

Hot meal

belong to ADL → 0.5: hot meal → 0.5: cold meal → 0.0: tea Sensor Event Freezer

&

→ 0.5: hot meal → 0.0: cold meal → 0.5: tea

slide-30
SLIDE 30

P01 D12 AD1C AD1B I05 I03 M19 M01 M23 M04 M03 M02 M05 M06 M12 M07 M11 M10 M09 M08 M13 M14 M15 M16 M17 M18 M21 M22 M24 M51 D11

Living / Dining Room Kitchen Storage Closet Closet

D10 D08 D09 I08 I09 I08

I01 I02 I04 I06 I07 D07

slide-31
SLIDE 31

PerCom ‘18, Athens, Greece, March 21, 2018

14.09.2016

31

MLN Model (detailed)

time-aware inference temporal knowledge-based

Ontological constraints

*SemanticCorrelation (SenEvent, ADL, ActivClass, p) *Event (SenEvent, EventType, Time) *InstanceCandidate (ADL, Start, Stop)

Observed predicates

OccursIn (SenEvent, ADL) InstanceClass (ActivClass, ADL)

Hidden predicates PPM Matrix *SemanticCorrelation Statistical analysis of events *InstanceCandidate / *Event

slide-32
SLIDE 32

PerCom ‘18, Athens, Greece, March 21, 2018

14.09.2016

32

Semantic Integration Layer

  • collects events data from a sensor network
  • applies preprocessing rules to detect operations

<Event(se1, et1, t1), …, Event(sek,etk,tk)>

fridge door sensor signaled “1” → the operation is “opening the fridge” Example

slide-33
SLIDE 33

PerCom ‘18, Athens, Greece, March 21, 2018

14.09.2016

33

Statistical Analysis of Events

Input: PPM matrix and temporally ordered events infers most probable activity class for each event allows to define activity boundaries (activity instance candidate) activity instance candidate Events Temporal extension

  • f MLN (MLNNC )

Knowledge Base