A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE - - PowerPoint PPT Presentation

A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE AND MOTOR FUNCTION IN THE ELDERLY

• PROF. KAT AGRES, NATIONAL UNIVERSITY OF SINGAPORE
• PROF. SIMON LUI, & TENCENT MUSIC
• PROF. DORIEN HERREMANS, SINGAPORE UNIVERSITY OF TECHNOLOGY AND DESIGN

IEEE CONFERENCE ON GAMES, LONDON, 2019

BACKGROUND

TECHNOLOGY FOR THE ELDERLY

Our aging population

• North America and Europe: > 1 in 5 people 


is > 60 years in 2017 (UN, 2017)

• Globally, the number of people >80 yrs is expected to increase more than 3 times by

2050, rising from 137 million in 2017 to 425 million

• High rates of population aged 65 and above include Japan (26.3%), Italy (22.4%),

Germany (21.2%), France (19.1%), and the United Kingdom (17.8%)

• Decreasing old-age support ratio

=> increasingly large burden on already-stressed health care systems across the globe.

What can we do about this?

• What strategic measures can be taken in preparation? How can

technology play a central role in preventive medicine? And how can we support the elderly in maintaining their motor and cognitive functions?

• Preventive medicine, e.g. music-based eHealth technologies

=> Prototype music system to improve cognitive and physical health and wellness for the elderly.

Existing work

• Since 2008: steady rise of games for health care, but few for older adults.
• Games for improving cognition in the elderly :
• Meta review 20 studies (1986-2013) (Toril, 2014): positive effect on multiple cognitive

functions such as memory, reaction time (RT), attention, and global cognition.

• Mostly existing games, e.g Luminosity (Brain games), real-time strategy games etc.
• Games for preventing physical decline:
• Physical exercises for motor rehabilitation in stroke patients (Chang, 2013; Agres, 2017)
• Balance rehabilitation (Lange, 2011)
• Improving balance in the elderly (Lai, 2013)

Integrating music

• Music therapy: effective in supporting both cognition and mental health:
• Improve a variety of cognitive functions, including executive control (Thaut, 2009),

attention (Kim, 2008), verbal recall (Gfeller, 1983; Wolfe, 1993), memory (Prickett, 1991; Simmons, 2010; Irish,

2006).

• Practicing music improve nonverbal memory and executive processing (Hanna 2011)
• Musical interventions significantly decrease anxiety & depression in the elderly (Thaut 2009)
• In-person music therapy is expensive (vs. eHealth tech.)
• Music can motivate people and increase compliance

=> We integrate music in our game to help prevent motor & cognitive decline in the elderly.

PROTOTYPE GAME

Game play

• The user hears a melody comprised of a series of four excerpts
• Each excerpt played by a different instrument
• When a particular instrument plays, the cartoon character holding that instrument on

the screen moves, to ensure that the user always knows which instrument is playing.

• Objective: remember the correct order of instruments, and indicate that order by

using the appropriate series of gestures.

• Recall memory tasks has been shown to contribute to the maintenance and

improvement of cognitive functioning in at-risk elderly people (Ngandu, 2015)

Smart gesture detection

• Kinect sensor with the Microsoft SDK & Unity
• Shown to parallel the functionality of a full multi-camera 3D motion system in

clinical settings (Clark, 2012)

• Rule based model to detect if correct instrument is being mimicked.
• Why these 4 instruments? They require:
• 1. Processing and remembering how each instrument is played
• 2. Bilateral movements that require coordination 


(e.g. bringing hands together for playing trumpet)

• 3. Each instrument: distinct movements from the user.

Smart gesture detection

System architecture

Demo

Customized music

• Music was composed to fit our requirements
• Four two-bar long phrases, each played by a different instrument


• The order of instruments is randomly varied across musical sequences,

and every instrument occurs exactly once in the sequence.

Conclusions

We have created a prototype serious game for the elderly that aims to:

• 1. leverage the health-related aspects of music 

• 2. increase motivation to remain physically active in old age

• 3. encourage the elderly to make coordinated, bilateral upper limb exercises

• 4. support cognitive function (interactive memory task)

The prototype is part of a larger suite of games for eHealth that includes games for strengthening and rehabilitation (Agres & Herremans, 2017; Beveridge, Cano, & Agres, 2018). Future work: personalised gesture detection, clinical trial, art design

SUTD Game Lab

Focus on serious games in the field of AI, education, healthcare, and AR/ VR technologies. The lab delivers commercially ready systems through collaboration with industry and faculty.

• Job opening: senior Unity developer
• Always looking for research collaborations in AI, healthcare, VR

games

• gamelab.sutd.edu.sg

A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE AND MOTOR FUNCTION IN THE ELDERLY

• PROF. KAT AGRES,
• PROF. SIMON LUI, &
• PROF. DORIEN HERREMANS

IEEE CONFERENCE ON GAMES, LONDON, 2019

DORIENHERREMANS.COM