Aging in Place Marjorie Skubic Associate Professor, Electrical and - - PDF document

3/31/2009 1

Recognition Technology for Aging in Place

Marjorie Skubic

Associate Professor, Electrical and Computer Engineering Dept. Director, Center for Eldercare and Rehabilitation Technology University of Missouri

Research Team

 MU Engineering

 Faculty: Marge Skubic, Jim Keller, Henry He, Harry Tyrer, Tina Smilkstein,

Tony Han

 Students/Staff: Derek Anderson, Tim Havens, Bob Luke, Isaac Sledge,

Shuang Wang, Fang Wang, Erik Stone, Kevin Reed, Nick Harvey, Wenqing Dai, Tanvi Banerjee, Dane Guevara, Chad Godsy, Matt Nevels, Ian Kable

 MU Nursing

 Faculty: Marilyn Rantz, Myra Aud, Bonnie Wakefield, Greg Alexander  Students: Jie Yu, Jean Krampe

 MU Health Informatics / Physical Therapy / Social Work

 Faculty: Mihail Popescu, Carmen Abbott, Deb Oliver, Colleen Galambos  Students: Saurav Garg , Rohan Ohol, Jarod Giger

 MU Family Medicine

 Richelle Koopman

 University of Washington

 Faculty: George Demiris

3/31/2009 2

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

TigerPlace

A new retirement community developed by the University of Missouri in affiliation with Americare Systems, Inc.

3/31/2009 3

The Challenge

Give us technology for TigerPlace that will help elder residents stay as active and functionally independent as possible.

Initial Focus Group Results

 Older adults were already using computers in

sophisticated ways

 Participants were willing to accept technology if it

 Met a need  Had an appropriate interface (watch sensory limitations)

 Wearable sensors were considered obtrusive  Privacy concerns with cameras were raised

 But participants said they might accept silhouette-based

cameras

Demiris, Rantz, Aud, Marek, Tyrer, Skubic & Hussam, Informatics and The Internet in Medicine, 2004.

3/31/2009 4

TigerPlace Designed for Aging in Place

11

TigerPlace Residents

 34 residents, aged from about 70 to 90  4 married couples + 26 singles + 2 dogs, 1 cat  90% have a chronic illness

 e.g., arthritis, heart disease, diabetes, stroke risk

 60% have multiple chronic illnesses  Some early stage Alzheimers  15% use a walker, 1 wheelchair, 1 in leg braces  Residents tend to be socially active

11

3/31/2009 5

Trajectory of Functional Decline

 Detect falls  Assess gait &

activity level

 Assess normal

patterns

 Recognize

pattern changes

 Detect acute

illness onset or changes

Time Current trend Aimed trend (with technology)

Functional Decline

Time

Rantz, Marek, Aud, Tyrer, Skubic, Demiris & Hussam, Nursing Outlook, 2005

Our Goal: Keep them functionally active!

3/31/2009 6

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

Design Decisions

 Do not put any sensors or devices on the person.  The system should be transparent to the elder.  Minimize changes to the environment.

3/31/2009 7

Integrated Sensor Network

16

Database Web server Residents, Family and Healthcare Providers Data monitor video sensors Physiological sensor network bed sensor Silhouette extraction Fusion + reasoning engine Customization manager stove sensor

Web interface

Alert manager motion sensors Activity analysis Activity analysis (1) (2)

(5) (4) (3)

(6) Video sensor network

Skubic, Alexander, Popescu, Rantz, Keller, Technology & Health Care, in press.

Basic Sensors

3/31/2009 8

Video Processing

Low cost cameras (on Tripods for the picture) Young Elliott Anderson

“Little Elliott”: Low cost CPU with GPU for processing power

$100 each $500 Priceless

A typical sensor network

14 motion sensors 1 Bed sensor unit 1 Stove temp unit 1 PC appliance

3/31/2009 9

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

Sensor Network Installations

21

Res.# 2005 2006 2007 2008 # Mo. 1 17 2 37 3 30 4 28 5 5 6 8 7 22 8 22 9 22 10 12 11 12 12 12 13 11 14 10 15 8 16 3 17 3

3/31/2009 10

Increase in Bed Restlessness Following Surgery

Heart attack and bypass surgery

Case Study #1

Rantz, Skubic, Miller, and Krampe, ICOST 2008

Return to Normal Bed Restlessness Following Cardiac Rehabilitation

Case Study #1

3/31/2009 11

Bradycardia (slow pulse rate: 1 -30 beats per minute)

ER visit Hospitalization

Case Study #2

Rantz, Skubic, Miller, and Krampe, ICOST 2008

Decrease in Bed Restlessness

ER visit Hospitalization

Case Study #2

3/31/2009 12

29

Motion Sensor Density Map

One resident in March later in October Each line is one day

Black indicates time out of apartment Color indicates density of motion activity Wang and Skubic, IE 2008

30

Two different residents

A more active pattern A pattern showing irregular activity

Wang and Skubic, IE 2008

3/31/2009 13

31

Combining Motion Sensor and Bed Sensor Data

 32 dimensional feature

vectors

 Bed Features: time

awake/asleep, # OOB morning/night, time in bed morning/night, nap time, time OOB morning/night, amount slept, stats. rest/pulse/breath firings

 Motion Features: # daily

bathroom visits, puttering index, time in each room, stats.

• f motion firings

PCA reduction

Sledge, Keller & Alexander, EMBC 2008

32

A Temporal View

3/31/2009 14

Growing Neural Gas for Temporal Clustering

33

10/10/2005 to 01/18/2006 10/10/2005 to 06/17/2006 10/10/2005 to 11/14/2006 Sledge & Keller, ICPR 2008

Interview Results

 No privacy concerns with motion and bed monitoring  Some discomfort with bed sensor  Three stages in adjustment to sensor network

 Familiarization (2-3 weeks)  Adjustment and curiosity (2-3 weeks)  Integration: participants forget about the sensors after

about one month and have not changed their activities

 Residents have asked to see the data themselves  Residents participate because they are helping us

34

Demiris, Parker Oliver, Dickey, Skubic, Rantz, Technology and Health Care, 2008.

3/31/2009 15

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

Use Silhouettes for Privacy

3/31/2009 16

Silhouette Improvement

Fuse color and texture

Original Image Color Feature Texture Feature Fusion (no post processing) Undetected Undetected Improvement

47

Results of Silhouette Extraction

Color/Texture Method RGB Method

Percentage of Foreground Pixels Found Percentage of Pixels Incorrectly Classified as Foreground Percentage of Foreground Pixels Found Percentage of Pixels Incorrectly Classified as Foreground Sequence 1 99% 9% 70% 7% Sequence 2 97% 5% 66% 6% Sequence 3 96% 8% 79% 6% Total 98% 7% 69% 6%

Color and Texture RGB

• nly

Luke, Anderson, Keller & Skubic, 2008.

3/31/2009 17

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

3/31/2009 18

Raw Video 1 Raw Video 2 Voxel Camera Space Voxel Lists Intersection Voxel Person

Creation of Voxel Person

Silhouette Extraction Silhouette Extraction

Silhouette 1 Silhouette 2

Voxel Space

3/31/2009 19

Voxel Person in Action Fuzzy Activity Analysis

INPUTS

• 1. Mean
• 2. Eigen based height
• 3. Max eigenvector and

ground plane normal similarity OUTPUTS Red = Upright Green = In between Blue = On the ground

Initial Implementation - 24 Fuzzy Rules

Anderson, Luke, Keller, Skubic, Rantz & Aud, IEEE TFS, 2009.

3/31/2009 20

Level 1: Fuzzy logic for state based reasoning … …

time t-1 time t time membership

S1 S2 S3 { Sum1 , Sum2 , … , SumG } Level 2: Fuzzy logic for reasoning about activity Voxel Person Silhouette Extraction Sum1 Sum2 SumG …

membership

Linguistic Summarizations … …

Linguistic Summarization

 <Person> is <State> in <Location> for <Time>

 where <State>, <Location>, and <Time> are

linguistic variables

60

LINGUISTIC SUMMARY EXAMPLES: Derek is upright in the living room for a moderate amount of time. Derek is on the ground in the living room for a moderate amount

• f time.

3/31/2009 21

Level 1: Fuzzy logic for state based reasoning … …

time t-1 time t time membership

S1 S2 S3 { Sum1 , Sum2 , … , SumG } Level 2: Fuzzy logic for reasoning about activity Voxel Person Silhouette Extraction Sum1 Sum2 SumG …

membership

Linguistic Summarizations … …

Fall Detection

 Second layer uses domain knowledge about

elderly falls

 Input  linguistic summaries  Output  confidence of a fall

62

Anderson, Luke, Keller, Skubic, Rantz & Aud, Computer Vision and Image Understanding, 2008

3/31/2009 22

False Alarms/Other Activities

Anderson, Luke, Keller & Skubic, WCCI 2008.

Fall Detection Results in the Lab

 16 short sequences (30-60 sec)

 12 falls total with some non-fall activities

 2 long sequences (7-11 min)

 2 falls total with lots of non-fall activities

 Each fall was successfully detected!  No false alarms reported!

Anderson, Luke, Skubic, Keller, Rantz & Aud, Gerontechnology, 2008.

3/31/2009 23

66

Acquiring Realistic Data Using Stunt Actors

But first, we had to train them!

Rantz, Aud, Alexander, Wakefield, Skubic, Luke, Anderson & Keller, Journal of Nursing Care Quality, 2008.

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

3/31/2009 24

Markerless Motion Capture

Estimating the Pose Tracking movement

• Dimensions of limbs and torso

and joint locations can be predefined to a generic human

• r fit to a specific subject’s

anthropometric measurements.

• Limbs and torso are

fleshed out with tapered superquadratic ellipsoids.

• Neck and wrists are 2-DOF

joints.

• Elbows and knees are 1-DOF

joints.

• Shoulders and hips are

3-DOF joints.

• Each limb is represented by a

kinematic chain and anchored to the torso

The Generic Human Model – 32 DOF

3/31/2009 25

Matching a Generic Human Model to the Silhouette

GPU Implementation makes it go fast!

Computational Elements of Exercise Motion Capture System

Video Sequences

• 640x480 pixels
• 24-bit RGB images
• 7.5 frames per second
• Image processing
• Background subtraction
• Morphological filtering
• Silhouette weighting

Silhouettes

• 320x240 pixels
• 8-bit scalar map
• 7.5 frames per second

3D Pose Estimate

A hybrid computational architecture that uses both the CPU and the Graphics Processing Unit (GPU) allows us to process

• ne second of video in approximately one minute of

computation.*

* Two camera views, 7.5 fps per camera

• Hierarchical particle

swarm

• 13 shape primitives
• ~200 vertices per shape
• ~400 triangles per shape
• 28 degrees of freedom
• Model articulation
• Model rasterization

Human Model

3/31/2009 26

Markerless Motion Capture in an Exercise Environment

 Overhead pull exercise

Silhouettes 3D Motion Model

Exercise Safety on a Treadmill

3/31/2009 27

2D Contour Tracking

Alexander, Havens, Skubic, Rantz, Keller & Abbott, Gerontechnology, 2008. Havens, Alexander, Abbott, Keller, Skubic & Rantz, IEEE CIVI 2009.

Roach Infestation Optimization

Tracking Spine Movement

• Less deviation
• Greater deflection angle
• Shows limp

3/31/2009 28

Tracking Shoulder Movement

78

Balanced, normal walk Limp with an offset

Spine Tracking Validation

Motion capture marker Motion capture marker Motion capture marker

3/31/2009 29

Spine Tracking Validation

100 200 300 400 500 600 700 800 900

• 10

10 20 30 40 50 60 Angle (deg) 100 200 300 400 500 600 700 800 900

• 10

10 20 30 40 50 60 70 80 90 Angle (deg)

Error Contour tracking Vicon System

Mean error = 2.2 deg

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

3/31/2009 30

HCC Project: Tracking Two People

View #1 View #2

HCC Voxel Data

89

3/31/2009 31

Participants

 10 participants – 2 male, 8 female; all over 65  Each participant acted out two scenarios 6 times

each  120 data runs

 Visitor scenario  Housekeeper scenario

 Imagery processed to extract silhouettes and

voxel representations

90

Interviews on Silhouette Imagery

 Investigate privacy issues  Semi-structured interviews, audio-taped

 Show residents their silhouettes and voxel data  Start with a list of questions but the order may vary  Make it a collaboration:  “…we want to know what you think”

 Terminology makes a difference - don’t call

them cameras!

91

3/31/2009 32

Assessing Impressions

 Now that you have seen this, what is your reaction?  If a system allows detection of emergencies and potentially

prevent accidents or emergencies, would you agree to having a video sensor like this in your apartment?

 What concerns would you have? (cues: privacy, autonomy)  Who would you like to have access to the data from such a

system? Would you like to see such images of yourself regularly?

 Would you like to have control when the system works and

when it does not?

 Would it bother you if you had this system in your apartment

and had friends or family visit you?

93

Content Analysis

 Coded by two investigators according to the

four dimensions of Bellotti and Sellen’s framework (1993)

 Capture - when and what information gets recorded  Construction - what happens to the information  Accessibility - who has access to the information  Purposes - how will the information be used

94

Bellotti V & Sellen A. (1993). “Design for privacy in ubiquitous computing environments,” Proc. 3rd European Conf. on Computer Supported Cooperative Work,

• G. de Michelis, C. Simone and K. Schmidt (Eds.), Kluwer, pp. 77-92.

3/31/2009 33

Interview Results

 Capture

 Participants found the silhouette images to be

protective of their privacy

 Participants wanted some control – ability to turn it

• ff

 Construction

 Most participants liked the silhouette imagery  They were less enthusiastic about the voxel data, as

they could not see the usefulness

95

Demiris, Parker Oliver, Giger, Skubic & Rantz, Medical Informatics & the Internet in Medicine, in review

Interview Results

 Accessibility

 Nine participants emphasized that access should be

limited to those who needed it for monitoring – not necessarily family members

 Seven participants would want to see their own data

 Purposes

 Seven participants saw the value for monitoring

although stated they personally did not need it

 One participant did not want it  One participant who had fallen recently did want it

96

3/31/2009 34

Outline

 Motivation

 TigerPlace

 Our Solution: Integrated sensor network  TigerPlace Installations  Development of Video Sensors

 Privacy protection  Detection of falls  Linguistic summarization  Markerless motion capture  HCC project

 Conclusions

 What do the residents think?

Conclusions

 19 systems installed; evaluation continues  Initial emphasis on one-person homes

 Algorithms to identify when resident is alone or with visitors and

detect changes in patterns

 Looking for sensor correlations with medical records

 HCC grant addresses multi-person environments

 Investigate privacy concerns using vision sensing  Monitor gait and physical function

 Much room for improvement

 New sensors, better processing, more sophisticated reasoning

3/31/2009 35

Conclusions about Residents

 Elders take ownership of the sensor data  Elders want control over who has access to their data  Acceptance is related to need and perceived benefits.  Privacy can be sacrificed for needs/benefits  Elders tend to underestimate their own needs  Elders care about the look of the technology  Elders are willing to accept technology if it

 Meets a need  Has an appropriate interface (address sensory limitations)

106

Research Funding

 NSF  ITR grant IIS-0428420  HCC grant IIS-0703692  U.S. Administration on Aging (90AM3013)  RAND/Hartford Foundation (9920070003)  NIH (NIH-5R21AG026412-02)

See http://eldertech.missouri.edu for publications

107

3/31/2009 36

Our Mission: Keeping older adults functionally active (and productive!)

See http://eldertech.missouri.edu for publications