Semantic Reasoning and Verification for Ambient Assisted Living - - PowerPoint PPT Presentation

▶
semantic reasoning and verification for ambient assisted
SMART_READER_LITE
LIVE PREVIEW

Semantic Reasoning and Verification for Ambient Assisted Living - - PowerPoint PPT Presentation

Sep 27, 2023 265 likes •728 views

Semantic Reasoning and Verification for Ambient Assisted Living Thibaut Tiberghien, Mounir Mokhtari, Liu Yan & Dong Jin Song FSFMA 2014 2nd French Singaporean Workshop on Formal Methods and Applications 13 May 2014, Singapore

slide-1
SLIDE 1

Semantic Reasoning and Verification for Ambient Assisted Living

Thibaut Tiberghien, Mounir Mokhtari, Liu Yan & Dong Jin Song

  • FSFMA 2014

2nd French Singaporean Workshop on Formal Methods and Applications 13 May 2014, Singapore

slide-2
SLIDE 2

Outline

  • I. Introduction
  • II. Context reasoning
  • Reasoning strategies
  • Reasoning architecture
  • III. Validation & Verification

IV.Conclusion

2

slide-3
SLIDE 3
  • Major demographic changes *

Population ageing

2010

10 20 30 40 50 60 70 80

  • ver 85

Males Females

2050

10 20 30 40 50 60 70 80

  • ver 85

Males Females 0% 2% 4% 6% 2% 4% 6% 0% 2% 4% 6% 2% 4% 6% Age Age

3

* United Nations, World Population Prospects: The 2010 Revision. http://esa.un.org/unpd/ wpp/index.htm

slide-4
SLIDE 4
  • Elderly support ratio is shrinking
  • e.g. in Singapore *

Population ageing

From 13.5 in 1970 to 2.6 in 2015!

4

* Resident Old-Age Support Ratio. Department of Statistics Singapore. http://www.singstat.gov.sg

slide-5
SLIDE 5

Progression of dementia

  • Projections of frail elderly population in Singapore *

5

* Dr. Philip Yap, Khoo Teck Puat Hospital, Singapore

Number of frail elderly

5000 15000 25000 35000 45000 2000 2010 2020 2030

7 000 beds available in nursing homes Need in 2015: 22 000 beds

slide-6
SLIDE 6

Sensing Context-aware framework Service provision

What is AAL?

6

slide-7
SLIDE 7

Scientific challenges

7

Sensing Context-aware framework Service provision

« Sensor prototyping » « Micro context acquisition » « Context awareness » « Reasoning techniques » « Modelling » « Service platform » « User interface plasticity » « Human factors »

Communication layer System verification

« Rules verification » « Information consistency » « Wireless communication » « Discovery protocols » « Privacy »

slide-8
SLIDE 8

Scientific challenges

8

Sensing Context-aware framework Service provision

« Sensor prototyping » « Micro context acquisition » « Context awareness » « Reasoning techniques » « Modelling » « Service platform » « User interface plasticity » « Human factors »

System verification

« Rules verification » « Information consistency » « Wireless communication » « Discovery protocols » « Privacy » collaboration with Liu Yan, Dong Jin Song, NUS (PAT)

  • ur research work

Communication layer

slide-9
SLIDE 9

Outline

  • I. Introduction
  • II. Context reasoning
  • Reasoning strategies
  • Reasoning architecture
  • III. Validation & Verification

IV.Conclusion

9

slide-10
SLIDE 10

Related work

10

Rule-based (Semantic Web) Pattern matching (HMM, DBN…) Sensor fusion and binding to context Detection of patterns in sequence of preprocessed sensor data + no prior knowledge data driven

  • labelled dataset for learning

fine grain sensing + leverage prior knowledge no learning

  • no implicit patterns

sensitive to noise

slide-11
SLIDE 11

Semantic Web Technologies

  • Metadata-data graph model
  • Make statements about resources
  • Triple-based representation
  • URI and namespaces
  • RDF, RDF-S, OWL (TBox)
  • Own namespaces (TBox + Abox)
  • Serializations: XML, N3, ...

11

« subject predicate object »

slide-12
SLIDE 12

Functional context model

12

Legend

datatypeProperty

Class

  • bjectProperty

subClassOf

stageForAlert isAlone inRoomSince inRoomFor doesActivitySince doesActivityFor

Resident

liveIn/detectedIn/cameFrom

motionMeasured

Environment

lastUsed

Object

hasValue lastUpdate hasLastUpdate

Sensor

indicateLocation indicateUse

SensorState Furniture Door SensorType House Room Outside Bedroom Bathroom Livingroom Kitchen Toilet ...

useNow locatedIn partOf type hasPossibleState/ hasCurrentState attachedTo deployedIn

name snoozeTime

believedToDo

name needHands getConfidenceScore

Context

busy

Caregiver Location Activity

solved riskLevel

Deviance Reminder

ackHandled acknowledgement

Notification

name model repeat id timeSent stage

Service

name

Device

watchesAfter runningFor solvedBy helpsWith usedBy escalateTo hasAckService deployedIn

  • nDevice

Person

slide-13
SLIDE 13

Rule design: epistemology

  • Type I: Rules of reasoning
  • Irrevocable knowledge
  • Scientific models, common sense
  • Type II: Rules of fact
  • Domain knowledge
  • Experts’ experience, intuition and insight
  • Possible contradiction (scoring)
  • Type III: Arbitration rules

13

slide-14
SLIDE 14

Type I - Rules of reasoning

14

∀ Sensor se; SensorState st; Room r, r2; House h; Resident u; Timestamp t;

  • {se hasCurrentState st} ⋀ {se hasLastUpdate true} ⋀ {st indicateLocation true}

⋀ {se deployedIn r} ⋀ {r partOf h} ⋀ {u liveIn h} ⋀ {u detectedIn r2} ⋀ {r2 ≠ r} ⋀ {se lastUpdate t} ⟹ {u detectedIn r} ⋀ {u cameFrom r2} ⋀ {u inRoomSince t}

slide-15
SLIDE 15

Type II - Rules of fact

15

slide-16
SLIDE 16

Type III - Arbitration rules

Compute the Rule-Based Confidence Score

16

slide-17
SLIDE 17

Outline

  • I. Introduction
  • II. Context reasoning
  • Reasoning strategies
  • Reasoning architecture
  • III. Validation & Verification

IV.Conclusion

17

slide-18
SLIDE 18

« La Pensée »

A Cognitively Inspired Hybrid Reasoning Architecture

18

Cogitation (conscious mind) Cerebration (unconscious mind) Knowledge formalisation

S t i m u l u s ( ~ e v e n t ) Cerebral cortex

slide-19
SLIDE 19

Live event processing

19

Legend invoke service exchange data

model.n3 home.n3 rules.n3 dump.n3 Stimulistener Cortex Cerebration* Cogitation EYE Decision Interpreter NTriplestore 1.live event 0.loaded once when ubismart starts 2.update(live event) 3.think i.think iv.think ii.query iii.update v.dump vii.think x.update ix.inferred triples vi.ontological dump at each inference viii.loaded in memory at each inference 5.decision xi.decision

slide-20
SLIDE 20

Hybrid reasoning

20

Service Delivery Modules Sensing Modules Configuration Modules Reasoning Modules "X" SensorFlowManager (Routing)
  • merge all "X" signals
  • registration of SensApps
  • forward data to relevant SensApp
  • forward information to InfoConsumers
  • discovery of new sensors
  • configuration gate for Binder
  • sensors to SensApps bindings
SensApp*
  • process data
  • produce information
BinderUI* SensAppRepository*
  • list SensApps per category of sensor
  • provide SensApp for download
"X" Gateway*
  • low level
  • protocol dependent (code is specific)
  • normalization of data (e.g. HEX2DEC)
  • signal forwarding

D I K W?

  • flow
  • no meaning
  • mostly useless
  • in the semsea (ontology)
  • semantic (meaning)
  • useful & calculable
SensApp dev. > introduce libs in SensApp API > restructure data in a more meaningful way using processes INFERENCE > process is not necessary semantic but I/O goes through semantic sea Binder
  • configuration tool (build, declarative)
  • choice, download & installation of SensApps
  • parametrization of SensApps
  • bind semantics (to sensor + its events)
Cortex
  • handle reasoning cycle (until stable cogitative state)
  • alternate between cogitation pass & cerebration pass
start/stop & feedback REST + Event Bus handleEvent processEvent processThoughts REST + download Legend SmartSpaceComposer (S2C): configuration * this module can have several instances this module is semantic-aware, i.e. linked with NTriplestore External Module Internal Module Abstract Class EventBridge* XMPPBridge DPWSBridge HTTPBridge OSGiBridge publish subscribe Stimulistener
  • match sensor/state with semsea bindings
  • write context information in semsea
subscribe NTriplestore
  • API to semantic sea (semsea)
  • provide CRUD access to the triplestore
  • implemented as a service
think Cogitation
  • high level rule-based reasoning
  • decision making
  • conscious process
Cerebration*
  • threaded asynchronous inference
  • low level, specific, computational task
  • I/O in seamsea
  • subconscious process
Classification Statistical Analysis Numeric Computation CoreEngine* EYE SVM think DecisionInterpreter
  • interpret thoughts (reasoning results)
  • update triplestore
  • start/stop service
  • etc.
ServiceGateway
  • registration of services
  • control over services
  • status update in triplestore
start/stop/update UbiService* BackEndService* FrontEndService* AtomicService* CoupledService* Reminder* CallCenter* ActivityLog* Notification* display(<ServiceSkin>) & ack register DeviceGateway
  • registration & management of devices
  • semsea I/O
register ComLayer* XMPP APNs UbiDevice* IPhone* IPTV* send & ack ServiceSkin* MobileGUI* TextToSpeech* ...
  • fused information
  • domain knowledge
  • higher level
subscribe InfoConsumer*
  • platform or application using the same data
slide-21
SLIDE 21

Ontology evolution breakdown

  • initial ontology

21 Legend

  • bject

datatype property

litteral

  • bject
  • bject property

update query

johndoe

  • ccupied

livingroom toilet house pir2 pir2_off pir6 pir6_on

believedToDo detectedIn partOf liveIn deployedIn deployedIn hasCurrentState hasCurrentState motion partOf

7

motion motion

7

id

A2

id

A6

hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 08:42:27

slide-22
SLIDE 22

Legend

  • bject

datatype property

litteral

  • bject
  • bject property

update query

johndoe

  • ccupied

livingroom toilet house pir2 pir2_on pir2_off pir6 pir6_on

believedToDo detectedIn partOf liveIn deployedIn deployedIn hasCurrentState hasCurrentState motion partOf

7

motion motion

7

id

A2

id

A6

hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 13:08:08

hasCurrentState

Ontology evolution breakdown

  • Stimulistener: new event update

22

slide-23
SLIDE 23

Ontology evolution breakdown

  • Cerebration: motion estimation

23 Legend

  • bject

datatype property

litteral

  • bject
  • bject property

update query

johndoe

  • ccupied

livingroom toilet house pir2 pir2_on pir6 pir6_on

believedToDo detectedIn partOf liveIn deployedIn deployedIn hasCurrentState motion partOf

7

motion

1

motion

8

id

A2

id

A6

hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 13:08:08

hasCurrentState

slide-24
SLIDE 24

Ontology evolution breakdown

  • Cogitation: activity inference

24

{?u qol:detectedIn ?r. ?r a qol:Bathroom. ?u qol:inRoomFor ?d. ?d math:lessThan 1800. ?a1 a qol:Hygiene. ?u qol:canDo ?a1} => {?a1 qol:getScore 9}.

Legend

  • bject

datatype property

litteral

  • bject
  • bject property

update query

johndoe

  • ccupied

hygiene livingroom toilet house pir2 pir2_on pir6 pir6_on

believedToDo cameFrom partOf liveIn deployedIn deployedIn hasCurrentState motion partOf

7

motion

1

motion

8

id

A2

id

A6

hasLastUpdate

2012-06-27 13:07:58

hasLastUpdate

2012-06-27 13:08:08

believedToDo detectedIn hasCurrentState

slide-25
SLIDE 25

Outline

  • I. Introduction
  • II. Context reasoning
  • Reasoning strategies
  • Reasoning architecture
  • III. Validation & Verification

IV.Conclusion

25

slide-26
SLIDE 26

Traditional AAL validation

  • Difficulty to obtain relevant ground truth
  • Synthetic dataset
  • Role-played and labelled dataset (living lab)
  • Irrelevant to real settings: is system usable?
  • Does not validate economical "performance"
  • Does not certify system behaviour

26

slide-27
SLIDE 27

A 3-fold validation process

  • Performance: Load test
  • Usability: In-situ system trial
  • STARhome: controlled (technical validation)
  • Singapore nursing home: semi-controlled (top-down)
  • French individual homes: no control (bottom-up)
  • Certification: Rules verification

27

slide-28
SLIDE 28

Performance

  • Reasoning load test to 250 houses (40K+ triples)

28

10 20 30 40 50 60 1 (344) 16 (2804) 31 (5264) 46 (7724) 61 (10184) 76 (12644) 91 (15104) 106 (17564) 121 (20024) 136 (22484) 151 (24944) 166 (27404) 181 (29864) 196 (32324) 211 (34784) 226 (37244) 241 (39704)

EYE Reasoning Duration (s) Nb of houses (Nb of triples) 34s 200 homes

slide-29
SLIDE 29

STARhome

29

Back-end

ZigBee gateway Wi-Fi router Tiny debian machine Server with web service repository Serial/USB LAN

slide-30
SLIDE 30

Nursing home in Singapore

30

  • 14 months trial
  • 8 residents with dementia
  • 2 caregivers
slide-31
SLIDE 31

Results

  • Recognition rate of events
  • Early detection of the health issues

31

events missed correct recognition events missed correct recognition

  • Feb. 2012 (phase 2)
  • Sep. 2012 (phase 3)

17% 29% 71% 83%

3 6 9 12 24/02/12 27/02/12 01/03/12 04/03/12 07/03/12 10/03/12 13/03/12

Nb of reminders per day

slide-32
SLIDE 32

Individual homes in France

32

Bedroom 1 Livingroom Bedroom 2 Kitchen WC Bath

door open/close sensor motion sensor internet-connected gateway (BeagleBone)

  • 3 homes
  • 2 months trial in average
slide-33
SLIDE 33

Results

  • Activity inference

33

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 00:00 06:00 12:00 18:00 00:00 Go out Nap Occupied Sleep time day

slide-34
SLIDE 34

Results

  • Contextual dashboard

34

2012-06-04 640 640 640 640 640 2012-06-05 824 824 824 824 824 2012-06-06 339 339 339 339 339 2012-06-07 1 2012-06-08 362 362 362 362 362 2012-06-09 781 781 781 781 781 2012-06-10 470 470 470 470 470 2012-06-11 543 543 543 543 543 2012-06-12 835 835 835 835 835 2012-06-13 388 388 388 388 388 2012-06-14 340 340 340 340 340 2012-06-15 585 585 585 585 585 2012-06-16 362 362 362 362 362 2012-06-17 371 371 371 371 371 2012-06-18 590 590 590 590 590 2012-06-19 409 409 409 409 409 2012-06-20 134 134 134 134 134 2012-06-21 274 274 274 274 274 2012-06-22 4 2012-06-23 394 394 394 394 394 2012-06-24 256 256 256 256 256 2012-06-25 288 288 288 288 288 2012-06-26 471 471 471 471 471 2012-06-27 422 422 422 422 422 2012-06-28 446 446 446 446 446 2012-06-29 518 518 518 518 518 2012-06-30 418 418 418 418 418 2012-07-01 191 191 191 191 191 2012-07-02 309 309 309 309 309 2012-07-03 250 250 250 250 250 2012-07-04 299 299 299 299 299 2012-07-05 1 2012-07-06 1 2012-07-07 1 2012-07-08 1 2012-07-09 42 42 42 42 42 2012-07-10 309 309 309 309 309 2012-07-11 172 172 172 172 172 2012-07-12 199 199 199 199 199

Outside Living room Kitchen Bedroom3 Bedroom2 Bedroom

Mon. Tue. Wed. Thu. Fri. Sat. Sun.

slide-35
SLIDE 35

Outline

  • I. Introduction
  • II. Context reasoning
  • Reasoning strategies
  • Reasoning architecture
  • III. Validation & Verification

IV.Conclusion

35

Liu Yan & Dong Jin Song (PAT) @Peacehaven Nursing Home

slide-36
SLIDE 36

Model checking of AAL system

  • People + environment + system modelling
  • Formalisms for concurrent interaction
  • Comprehensive verification for certification
  • Automatic verification and exhaustive search
  • Counter-examples useful for bug tracking
  • E.g. with rule conflict

36

slide-37
SLIDE 37

Formal analysis approach

37

Counterexamples PAT Model Checker Verification: Safety, Liveness Modelling: Environment, System Design Stakeholders: Nurses, Engineers Collecting Patient behaviours Collecting Requirements

slide-38
SLIDE 38

Labelled transition systems

  • Modelling of people’s behaviours

38

slide-39
SLIDE 39

Labelled transition systems

  • Modelling of system's behaviours

39

slide-40
SLIDE 40

Properties specification

  • P1: Deadlock freeness
  • P2: Guaranteed reminders
  • P3: System inconsistency
  • P4: Conflicted or erroneous reminders

40

#assert SmartBedroom() |= [](UsingWrongBed -> <>RemindedWrongBed); #define InConsistency (Pos_Person[1] == SHOWROOM && ShowerFlag && sensors[PIR] == SILENT); # define ConflictReminder (ReminderStage[ShowerNoSoap * 2] != 0 && ReminderStage[WanderingInSR * 2] != 0);

slide-41
SLIDE 41

Verification result

  • Run on Intel Xeon CPU E5506 2.13GHz, 32GB RAM
  • Total runtime around 3.5hours

41

Bugs? Property Result Nb of states Nb of transitions Execution time (s)

  • P1.1
  • OOM

J

P1.2 True 1,433,654 2,038,064 815

J

P1.3 True 10,783,353 15,832,370 7,045

J

P2.1 True 1,599,797 2,430,351 1,945

L

P2.2 False 68,178 130,734 39

L

P2.3 False 2,192,251 4,531,005 12,414

L

P2.4 False 832,144 1,663,779 729

L

P2.5 False 4,314 5,150 1.6

J

P2.6 True 1,579,579 2,377,381 1,913

L

P3 True 572 745 0.3

L

P4.1 True 14,675 20,615 6.1

L

P4.2 True 2,446 3,036 1.11

L

P4.3 True 2,332,744 3,001,756 1,047

slide-42
SLIDE 42

Verification result

42

System part # Rules # Non-reachable # Redundant/ #Duplicated # Conflict Bedroom 17 2 8/3 2 Shower Room 22 5 16/2

  • Avg. Time (s)
  • 2.05

3.05

  • System split in two for memory purpose
slide-43
SLIDE 43

Bug report with error trace

  • System inconsistency
  • enterShowerRoom.1 -> turnOnTap ->

exitShowerRoom.1 -> port.PIRShowerRoom.Silent

  • False alarm
  • enterBedroom.2 -> sitOnBed.2.1 ->

promptReminder.1

43

slide-44
SLIDE 44

Outline

  • I. Introduction
  • II. Context reasoning
  • Reasoning strategies
  • Reasoning architecture
  • III. Validation & Verification

IV.Conclusion

44

slide-45
SLIDE 45

Conclusion

  • Contribution of model checking to AAL
  • Formal verification and certification of system
  • Useful bug tracking tool at design level
  • Save time on costly deployments
  • Avoid patient disturbance of early testing
  • An excellent verification tool during system design
  • To be used in conjunction with real deployments

45