Antnio Teixeira, Nuno Almeida, Carlos Pereira, Miguel Oliveira e - - PowerPoint PPT Presentation

W3C MMI Architecture as a Basis for Enhanced Interaction for Ambient Assisted Living

António Teixeira, Nuno Almeida, Carlos Pereira, Miguel Oliveira e Silva ajst@ua.pt

Overview

• Our motivations

– AAL – Interaction

• …and goals
• Information on usage of MMI architecture in

AAL projects University of Aveiro/IEETA is involved

– AAL4ALL – Smartphones for Seniors – AAL Paelife

• Some movies and demo(s)

Motivations:

• Multimodal Interaction
• Speech centric
• Interaction for AAL

5

• Dialog
• Speech …

– Speech + gestures + expressions …

Human interaction

5

6

• Development of Robust,

Useful and Easy to Use Human- Machine Interaction

• Using speech and language,
• Supporting Portuguese native speakers

– We have been involved in spoken interaction since the 90s

Long term goal

6

AAL

Ambient Assisted Living

8

• Based on the recognition of the human rights of older people

and the United Nations (UN) principles of independence, participation, dignity, care and self-fulfillment;

• Is the process of optimizing opportunities for health,

participation and security in order to enhance quality of life as people age;

• Refers to continuing participation in social, economic, cultural,

spiritual and civic affairs

• Allows people to realize their potential for physical, social, and

mental well-being throughout the life course and to participate in society, while providing them with adequate protection, security and care when they need

Motivations - Active Ageing

9

• The introduction of technological solutions can

facilitate the daily life of the elderly, fighting isolation and exclusion, increasing their pro-activity, work capacity and autonomy.

• Addressing Accessibility

– There is a need to overcome the gap between an individual's capabilities and the technological resources,

• Addressing Usability

– a term used to denote the ease with which people can employ a particular tool or other human- made object in order to achieve a particular goal. Motivations – Tech Innovation

AAL and multimodality

• Strong focus on contextual usage, usability,

pervasiveness, and the user

– Leading to highly heterogeneous environments

• Multimodality can provide an important push in

shortening the gap between the user and its ambient

• AAL challenges:

– necessity of including a wide number of devices – distributed focus of AAL – requirement of constant connection

10

• The architecture has the potential to

solve some of our issues for interaction in AAL:

– Heterogeneity

• now a smaller issue

– Autonomy is increased – Usability through choice

W3C MMI architecture

Information on concrete uses

13

Example 1 - AAL4ALL

• AAL4ALL project has a specific task (under the

leadership of University of Aveiro) to define the architecture and develop multimodal support for AAL Applications

• W3C

Architecture Adoption within the task followed several phases – First phase

» Some experiments with EMMA » Creation of an MMI interoperability test report

– Second phase

» Tests with Interaction Manager based on SCXML (on the right)

– Third phase

» First ‘blackbox’ modalities supporting Lifecycle Events communication – ASR, TTS with Interaction Manager iteration » Test framework based only on EMMA and Lifecycle Events » First integrated demo

– Ongoing

» Developing and integrating more modalities (ex: avatar) » Work on fusion and fission capabilities

15

AAL4ALL Demonstrator - July 2012

• To prove the potential of W3C multimodal architecture, a

cooperation was made between University of Aveiro and Microsoft Language Development Center (in Lisbonne).

• Development of a simple tecnological demonstrator making

use of input and output modalities developed in scope of the project.

• The demo consisted in an

multimodal integrated water heater controller.

• It was presented at project

general meeting in July 2012, Porto, Portugal

16

AAL4ALL Demonstrator - July 2012

19

Example 2 – Smartphones for Seniors System Architecture

• Windows Phone 7
• Web services to overweight features
• Several modules to control modalities and

intelligence

20

Multimodal Output

• Graphics and Text

– Panoramic view – Medication images generation – Medication details

• Speech

– TTS (e.g. “The side effects of medication are…”) – Use of Natural Lang Generation

• Ex: “You should take one tablet of Aspirin”

21

The movie …

• Goal:

– Accessible and easy-to-use Assistant that integrates a set of services aiming at increasing social activity and social- integration of the elderly, making use of multimodal interaction.

Example 3 – PaeLife Personal Assistant

22

• Project adopted a system composed of

both a stationary main unit and a portable unit.

System architecture overview Platform

23

Main Unit Home computer Home computer Screen Screen PLA Database Portable Unit Portable Unit (Premium Tablet) Portable Unit (Premium Tablet) Portable Unit (Low Cost Tablet) Portable Unit (Low Cost Tablet) Home Office 365 Voice Search Automatic Speech Recognition

Internet

authentification services Text to Speech kinect

• Main unit integrates the

mentioned big screen

– supporting graphical output modalities and also sound and speech output

• Home computer
• And input devices such as

Microsoft Kinect

– providing the means for the body gestures input modality

System architecture overview Platform: main unit

24

• Integrated multi-touch

support in the operating system makes possible the touch input modality

• Integrated microphone

and speakers will enable the speech input and

• utput modalities
• The display will support

graphical output modalities

System architecture overview Platform: portable unit

25

• Since the PLA will be composed of two

units each one being independent yet simultaneously interoperable

• The following types of interaction are

possible:

– Only through the main unit – Only through the portable unit – Both through the main and portable units

Interaction with PLA

26

• The Paelife multimodal framework is

directly based on the “standards” defined by the W3C

– This choice is justified by the architecture’s open standard nature. – This architecture provides an answer to the previous requirements easing the creation and integration of new modules Architecture

27

• Speech input/output modalities were

already created

– Using Microsoft’s speech platform – Easy support for the several languages and future language packs for Elderly Speech ASR

• Input Modality for Body Gestures

– Uses the PaeLife Kinect Framework built for the PaeLife project for recognition of user gestures

Modalities

28

• To serve as a test for the produced

framework, a demo application was created.

• The application recreates a news reader.

Proof-of-concept demonstration of the Framework

29

DEMO DEMO

• This work presented is part of the COMPETE - Programa

Operacional Factores de Competitividade and the European Union (FEDER) under projects AAL4ALL (www.aal4all.org), Smartphones for Seniors (www.martphones4seniors.org) and Living Usability Lab for Next Generation Networks (http://www.livinglab.pt/).

• Authors also acknowledge the funding for project PaeLife

(www.paelife.eu) from AAL JP and national agencies: Part of the work presented was funded by FEDER, COMPETE and FCT in the context of AAL/0015/2009 and IEETA Research Unit funding FCOMP-01-0124-FEDER-022682 (FCT-PEstC/EEI/UI0127/2011).

• Special thanks to all our partners

Acknowledgments