A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE - - PowerPoint PPT Presentation

a novel music based game with motion capture to support
SMART_READER_LITE
LIVE PREVIEW

A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE - - PowerPoint PPT Presentation

A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE AND MOTOR FUNCTION IN THE ELDERLY PROF. KAT AGRES, NATIONAL UNIVERSITY OF SINGAPORE PROF. SIMON LUI, & TENCENT MUSIC PROF. DORIEN HERREMANS, SINGAPORE UNIVERSITY OF TECHNOLOGY


slide-1
SLIDE 1

A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE AND MOTOR FUNCTION IN THE ELDERLY

  • PROF. KAT AGRES, NATIONAL UNIVERSITY OF SINGAPORE
  • PROF. SIMON LUI, & TENCENT MUSIC
  • PROF. DORIEN HERREMANS, SINGAPORE UNIVERSITY OF TECHNOLOGY AND DESIGN

IEEE CONFERENCE ON GAMES, LONDON, 2019

slide-2
SLIDE 2

BACKGROUND

TECHNOLOGY FOR THE ELDERLY

slide-3
SLIDE 3

Our aging population

  • North America and Europe: > 1 in 5 people 


is > 60 years in 2017 (UN, 2017)

  • Globally, the number of people >80 yrs is expected to increase more than 3 times by

2050, rising from 137 million in 2017 to 425 million

  • High rates of population aged 65 and above include Japan (26.3%), Italy (22.4%),

Germany (21.2%), France (19.1%), and the United Kingdom (17.8%)

  • Decreasing old-age support ratio

=> increasingly large burden on already-stressed health care systems across the globe.

slide-4
SLIDE 4

What can we do about this?

  • What strategic measures can be taken in preparation? How can

technology play a central role in preventive medicine? And how can we support the elderly in maintaining their motor and cognitive functions?

  • Preventive medicine, e.g. music-based eHealth technologies

=> Prototype music system to improve cognitive and physical health and wellness for the elderly.

slide-5
SLIDE 5

Existing work

  • Since 2008: steady rise of games for health care, but few for older adults.
  • Games for improving cognition in the elderly :
  • Meta review 20 studies (1986-2013) (Toril, 2014): positive effect on multiple cognitive

functions such as memory, reaction time (RT), attention, and global cognition.

  • Mostly existing games, e.g Luminosity (Brain games), real-time strategy games etc.
  • Games for preventing physical decline:
  • Physical exercises for motor rehabilitation in stroke patients (Chang, 2013; Agres, 2017)
  • Balance rehabilitation (Lange, 2011)
  • Improving balance in the elderly (Lai, 2013)
slide-6
SLIDE 6

Integrating music

  • Music therapy: effective in supporting both cognition and mental health:
  • Improve a variety of cognitive functions, including executive control (Thaut, 2009),

attention (Kim, 2008), verbal recall (Gfeller, 1983; Wolfe, 1993), memory (Prickett, 1991; Simmons, 2010; Irish,

2006).

  • Practicing music improve nonverbal memory and executive processing (Hanna 2011)
  • Musical interventions significantly decrease anxiety & depression in the elderly (Thaut 2009)
  • In-person music therapy is expensive (vs. eHealth tech.)
  • Music can motivate people and increase compliance

=> We integrate music in our game to help prevent motor & cognitive decline in the elderly.

slide-7
SLIDE 7

PROTOTYPE GAME

slide-8
SLIDE 8

Game play

  • The user hears a melody comprised of a series of four excerpts
  • Each excerpt played by a different instrument
  • When a particular instrument plays, the cartoon character holding that instrument on

the screen moves, to ensure that the user always knows which instrument is playing.

  • Objective: remember the correct order of instruments, and indicate that order by

using the appropriate series of gestures.

  • Recall memory tasks has been shown to contribute to the maintenance and

improvement of cognitive functioning in at-risk elderly people (Ngandu, 2015)

slide-9
SLIDE 9

Smart gesture detection

  • Kinect sensor with the Microsoft SDK & Unity
  • Shown to parallel the functionality of a full multi-camera 3D motion system in

clinical settings (Clark, 2012)

  • Rule based model to detect if correct instrument is being mimicked.
  • Why these 4 instruments? They require:
  • 1. Processing and remembering how each instrument is played
  • 2. Bilateral movements that require coordination 


(e.g. bringing hands together for playing trumpet)

  • 3. Each instrument: distinct movements from the user.
slide-10
SLIDE 10

Smart gesture detection

slide-11
SLIDE 11

System architecture

slide-12
SLIDE 12

Demo

slide-13
SLIDE 13

Customized music

  • Music was composed to fit our requirements
  • Four two-bar long phrases, each played by a different instrument



 
 


  • The order of instruments is randomly varied across musical sequences,

and every instrument occurs exactly once in the sequence.

slide-14
SLIDE 14

Conclusions

We have created a prototype serious game for the elderly that aims to:

  • 1. leverage the health-related aspects of music 

  • 2. increase motivation to remain physically active in old age

  • 3. encourage the elderly to make coordinated, bilateral upper limb exercises

  • 4. support cognitive function (interactive memory task)

The prototype is part of a larger suite of games for eHealth that includes games for strengthening and rehabilitation (Agres & Herremans, 2017; Beveridge, Cano, & Agres, 2018). Future work: personalised gesture detection, clinical trial, art design

slide-15
SLIDE 15

SUTD Game Lab

Focus on serious games in the field of AI, education, healthcare, and AR/ VR technologies. The lab delivers commercially ready systems through collaboration with industry and faculty.

  • Job opening: senior Unity developer
  • Always looking for research collaborations in AI, healthcare, VR

games

  • gamelab.sutd.edu.sg
slide-16
SLIDE 16

A NOVEL MUSIC-BASED GAME WITH MOTION CAPTURE TO SUPPORT COGNITIVE AND MOTOR FUNCTION IN THE ELDERLY

  • PROF. KAT AGRES,
  • PROF. SIMON LUI, &
  • PROF. DORIEN HERREMANS

IEEE CONFERENCE ON GAMES, LONDON, 2019

DORIENHERREMANS.COM