CUIML: A Language For the Generation of Multimodal Human-Computer - PowerPoint PPT Presentation

CUIML: A Language For the Generation of Multimodal Human-Computer Interfaces Christian Sandor sandor@cs.tum.edu Technische Universit¨ at M¨ unchen Chair for Applied Software Engineering

Abstract � DWARF Project at the Technische Universit¨ at M¨ unchen � UIML complied with some of our re- quirements � Extension of UIML was developed and will now be presented 2

What is DWARF? � Distributed Wearable Augmented Reality Framework � DWARF movie Augmented Reality Wearable Computer 3

The Problem � Framework for Presentation Layer of Wearable Computers • An application should be platform independent 4

The Problem � Framework for Presentation Layer of Wearable Computers • An application should be platform independent • Interaction with the user is multimodal Voice Input Actions trigger Mouse Input Actions 4

The Problem � Framework for Presentation Layer of Wearable Computers • An application should be platform independent • Interaction with the user is multimodal Voice Input Actions trigger Actions Mouse Input 5

CUIML – a solution � Cooperative User Interfaces Markup Language � Based on UIML • Separation of document structure and presentation • Transformation to markup languages • Display in browsers for VRML, VoiceXML, HTML (Views) � New concepts: • Controller • Manipulators 6

System Design – Views � UIML: • Adresses requirement of platform independence • No support for multiple cooperating Views UIML document Generic Structure A B C HTML Generation H1 D E Process H2 H3 Mapping H2 H2 HTML 1. A = H1 D = H2 B = H2 E = H2 C = H3 WML WML 2. W1 A = W1 D = W7 W2 W2 B = W2 E = W7 W7 W7 C = W2 7

System Design – Views (2) � Multimodal Human-Computer Interfaces consist of multiple Views CUIML document HCI 1 VRML HTML 1 3 Generic Structure 2 4 5 Generation A Process VoiceXML B C D 6 E F G H I 7 8 1. Mapping 1. VRML HTML VoiceXML C = 3 D = 6 B = 1 HCI 2 F = 4 H = 7 E = 2 G = 5 I = 8 VRML WML 2. 7 2. VRML WML 1 4 8 C = 1 D = 4 2 3 5 6 B = 7 F = 2 H = 5 E = 8 G = 3 I = 6 8

System Design – Controller � A central component is needed • Describes state of the HCI • Synchronizes views View View ?? View 9

System Design – Controller � A central component is needed • Describes state of the HCI • Synchronizes views � Deterministic Finite Automaton with XML based Configuration View View DFA View 10

System Design – Controller (2) � CUIML extended with Controller CUIML document HCI 1 VRML HTML Controller Config = 1 3 Generic Structure 2 4 5 Generation A Process B C D Controller E F G H I 1. Mapping VoiceXML 1. VRML HTML VoiceXML 6 C = 3 D = 6 7 8 B = 1 F = 4 H = 7 E = 2 HCI 2 G = 5 I = 8 2. VRML WML 2. VRML WML 7 1 4 C = 1 D = 4 B = 7 8 F = 2 H = 5 E = 8 2 3 5 6 G = 3 I = 6 Controller 11

Controller Example � State transitions of a View Printer available PrintJobQuestion Yes No Rejected Confirmed Controller VRML View 12

Controller Example � State transitions of a View Printer available PrintJobQuestion Yes No Rejected Confirmed Controller VRML View 13

Controller Example � State transitions of a View Printer available PrintJobQuestion Yes No Rejected Confirmed Controller VRML View 14

System Design – Manipulators � Views have to be modified by the Controller • Access mechanisms for Views differ View ?? Controller 15

System Design – Manipulators � Views have to be modified by the Controller • Access mechanisms for Views differ • Every View needs an appropriate Manipulator • Described in behaviour section View Manipulator Controller 16

System Design – Manipulators (2) � Example of a Manipulator CUIML document Controller Structure Generation A Process B C Manipulator D E A’ B’ C’ Controller Event "X" Behaviour D’ E’ Controller Event "X" => A Controller B C D E Manipulator A’ B’ C’ D’ E’ 17

Benefits of CUIML � Eases the task of developing multimodal HCIs � Dynamic reconfiguration of the HCI to preserve functionality � Incorperates the advantages of UIML • Separation of development tasks • Platform independence 18

Future Work � Rewrite renderers with JavaCC � Adapt the Human-Computer Interface at runtime 19

Future Work � Rewrite renderers with JavaCC � Adapt the Human-Computer Interface at runtime � Establish standard metaphors http://www.csl.sony.co.jp/person/rekimoto/cube/ http://www.cg.tuwien.ac.at/research/vr/pip/ 19