Interaction Management for Ubiquitous Augmented Reality User - - PowerPoint PPT Presentation

▶
interaction management for ubiquitous augmented reality
SMART_READER_LITE
LIVE PREVIEW

Interaction Management for Ubiquitous Augmented Reality User - - PowerPoint PPT Presentation

Aug 26, 2022 358 likes •747 views

Diplomarbeit Interaction Management for Ubiquitous Augmented Reality User Interfaces CAR - Car Augmented Reality Aufgabensteller: Prof. Gudrun Klinker Ph.D. Betreuer: Dipl.-Inform. Christian Sandor Vortragender: Otmar Hilliges May 13,

slide-1
SLIDE 1

May 13, 2004

CAR - Car Augmented Reality

Aufgabensteller:

  • Prof. Gudrun Klinker Ph.D.

Betreuer: Dipl.-Inform. Christian Sandor Vortragender: Otmar Hilliges

Diplomarbeit

Interaction Management for Ubiquitous Augmented Reality User Interfaces

slide-2
SLIDE 2

May 13, 2004 Otmar Hilliges 2/36

Summary

  • Diploma thesis within the CAR project November ‘03 - May

‘04.

  • Designed and implemented a method for interaction

management for UAR systems.

– Providing easy I/O device adaption – Introduced an abstraction layer for I/O devices. – A powerful formal model to design UI behavior.

  • Designed and implemented a runtime development

environment.

– Significantly decreases implementation of UIs (runtime prototyping). – Allows the adaption and exchange of devices at runtime. – Tweaking and tuning UI behaviour to experiment with interaction techniques is possible.

  • Implemented the UI behavior descriptions for CAR.
slide-3
SLIDE 3

May 13, 2004 Otmar Hilliges 3/36

Outline

  • Introduction
  • Requirements Analysis
  • Related Work
  • Implementation
  • Future Work
slide-4
SLIDE 4

May 13, 2004 Otmar Hilliges 4/36

Introduction

  • What are UAR user interfaces?
  • What is the problem space for such user

interfaces?

  • What design issues do those problems

precipitate?

slide-5
SLIDE 5

May 13, 2004 Otmar Hilliges 5/36

Introduction - Concepts

  • Ubiquitous Augmented Reality user interfaces

– Multi-user – Multi-device – Multi-modal – Mobile and distributed

slide-6
SLIDE 6

May 13, 2004 Otmar Hilliges 6/36

Introduction - Collaboration

Co-allocated vs. Collaborative working

slide-7
SLIDE 7

May 13, 2004 Otmar Hilliges 7/36

  • UAR user interfaces incorporate new devices

– Special purpose input devices. – Multimedia output.

Introduction - I/O adaption

slide-8
SLIDE 8

May 13, 2004 Otmar Hilliges 8/36

Introduction - Multimodal Integration

DWARF UIC DWARF UIC

slide-9
SLIDE 9

May 13, 2004 Otmar Hilliges 9/36

  • Variety of I/O devices
  • Dynamic system setups
  • Non standardized interaction techniques
  • Experiments with interaction techniques must be carried
  • ut
  • Changing the connectivity structure at runtime

Runtime Prototyping

Introduction - Runtime Prototyping

slide-10
SLIDE 10

May 13, 2004 Otmar Hilliges 10/36

Outline

  • Introduction
  • Requirements Analysis
  • Related Work
  • Implementation
  • Future Work
slide-11
SLIDE 11

May 13, 2004 Otmar Hilliges 11/36

Requirements Analysis

  • The requirements have been gathered

throughout different projects:

– TRAMP. – SHEEP. – ARCHIE – CAR.

slide-12
SLIDE 12

May 13, 2004 Otmar Hilliges 12/36

Requirements Analysis

  • Functional Requirements:

– Adapt I/O components. The control component is the glue that holds together the complete UI. – Input fusion. To deal with different modalities the component must be able to integrate multi-modal input. – Output fission. Generate content for multiple output components. – Input Recognition. Disambiguate input from inter-social communication. – Handle Privacy. Differentiate between public and private information. – Formal model to describe UI behavior is needed that can be executed, modified and stored persistently.

slide-13
SLIDE 13

May 13, 2004 Otmar Hilliges 13/36

Requirements Analysis

  • Non - Functional Requirements:

–

  • Availability. If the UIC fails the whole system gets unusable.

–

  • Robustness. New users will make errors in the usage of the

system. –

  • Reliability. The same interactions must always produce the

same results. –

  • Responsiveness. For usability reasons the user must get

immediate feedback whether an interaction succeeded or not. – Scalability due to steep increasing interpretation and management effort. – Flexibility to deal with inherently dynamic setups and changing I/O components.

slide-14
SLIDE 14

May 13, 2004 Otmar Hilliges 14/36

Requirements Analysis

  • Pseudo Requirements:

– DWARF is the target environment and the developed component must be able to communicate with other services.

slide-15
SLIDE 15

May 13, 2004 Otmar Hilliges 15/36

Outline

  • Introduction
  • Requirements Analysis
  • Related Work
  • Implementation
  • Future Work
slide-16
SLIDE 16

May 13, 2004 Otmar Hilliges 16/36

Related Work

  • Interaction Management

– Quickset – Unit – MetaDESK – Papier-Mâché – DART

  • Petri Net vs. Finite automata
  • Runtime Prototyping
slide-17
SLIDE 17

May 13, 2004 Otmar Hilliges 17/36

Related Work: Quickset

  • Quickset: Cohen et.al

Oregon Institue of Science and Technology System for collaborative, multi-modal planning of tactical military simulations. + Powerful integration of speech, gesture and web-based input. + Very robust resolving disambiguites using AI techniques.

  • Rigid architecture heavily application dependent.
  • System can not be used in other setups.

QuickTime™ and a Cinepak decompressor are needed to see this picture.

slide-18
SLIDE 18

May 13, 2004 Otmar Hilliges 18/36

Related Work: Unit

  • Unit: Alex Olwal, Columbia University 2002
  • Framework for the design of flexible interaction

techniques.

  • Abstraction layer between I/O devices and

application.

  • Units form a graph that allows the programmer

to develop powerful interaction techniques. + Flexible data manipulation. + Units are reusable.

  • No clear differentiation between discrete and

continous data.

  • Developers have to deal with I/O device‘s

details.

QuickTime™ and a Cinepak decompressor are needed to see this picture.

slide-19
SLIDE 19

May 13, 2004 Otmar Hilliges 19/36

Related Work: MetaDESK

  • MetaDESK: Brygg Ulmer et.al., MIT 1997

Groundbreaking system in the field of TUIs. The DESK is a illuminated table enriched with special purpose tools (TUIsf) for urban planning. + Lots of creative tangible interaction and presentation techniques.

  • Software architecture is application specific.
slide-20
SLIDE 20

May 13, 2004 Otmar Hilliges 20/36

Related Work: Papier-Mâché

  • Papier-Mâché:

A Toolkit for developing TUIs. Using computer vision, electronic tags and barcodes. + Provides a API for TUI based systems. + Includes a variety of out of the box recognition algorithms.

  • Code based approach.
  • Only focuses on TUIs.
slide-21
SLIDE 21

May 13, 2004 Otmar Hilliges 21/36

Related Work: DART

  • DART:

A toolkit for AR applications using a classic multimedia design tool (Macromedia Director). + Very easy to create content and application logic for non-programmers. + Director is already well-know and provides powerful means to design UIs.

  • Interactions are very limited.
  • Not changeble at runtime.
slide-22
SLIDE 22

May 13, 2004 Otmar Hilliges 22/36

RW: Petri Nets vs. Finite Automata

  • Petri Nets:

– Introduced to model concurrent and distributed systems. – Powerful mathematical model – Meets requirements for distributed, multi-user and multi-modal systems. – High ceiling

  • FNA:

– FNAs are used to model workflows (navigation, repair instructions). – One active state. Step by Step execution. – Very diffucult to model concurrent and multi-user situations. – Low learning threshold

slide-23
SLIDE 23

May 13, 2004 Otmar Hilliges 23/36

Related Work: Runtime Development

  • Squeak:

– Multimedia design and development environment for educational purposes. Fully tweak-able. – Very easy to develop interactive graphical

  • applications. Even kids can do it.

– Limited to the classic WIMP-desktop.

QuickTime™ and a MPEG-4 Video decompressor are needed to see this picture.

slide-24
SLIDE 24

May 13, 2004 Otmar Hilliges 24/36

Outline

  • Introduction
  • Requirements Analysis
  • Related Work
  • Implementation
  • Future Work
slide-25
SLIDE 25

May 13, 2004 Otmar Hilliges 25/36

Implementation

  • What I implemented in

this thesis:

– Interaction Management component based on DWARF and Petri Nets. – A runtime development environment for that component.

slide-26
SLIDE 26

May 13, 2004 Otmar Hilliges 26/36

Implementation

  • Layering and 3rd party software

– DWARF, Jfern, Graham-Kirby Compiler

slide-27
SLIDE 27

May 13, 2004 Otmar Hilliges 27/36

Implementation

  • Integration with DWARF UI architecture
slide-28
SLIDE 28

May 13, 2004 Otmar Hilliges 28/36

Implementation: Interaction Management

  • Multi-modal integration

– Input components emit tokens – Data is analyzed and modified inside Petri nets transitions – Commands are sent out to

  • utput components
slide-29
SLIDE 29

May 13, 2004 Otmar Hilliges 29/36

Implementation: Runtime Prototyping

  • Runtime development

– Net structure modifications – Dynamic code modification – Connectivity management

slide-30
SLIDE 30

May 13, 2004 Otmar Hilliges 30/36

Implementation: Runtime Prototyping

  • Results: Mini-Sheep and CAR UI
slide-31
SLIDE 31

May 13, 2004 Otmar Hilliges 31/36

Implementation: Object Design

  • UIC Implementation Details
  • Communication

– Handle service startup and communication – Receive and send structured events. – Query and Modify Needs & Abilities

  • Net Administration

– Execute Petri Net. – Add/Remove tokens. – Modify net structure. – Compile guards and actions.

  • GUI

– Visualize Petri net execution. – Controls for Editing PN and N&A. – Logging and debugging output.

slide-32
SLIDE 32

May 13, 2004 Otmar Hilliges 32/36

Outline

  • Introduction
  • Requirements Analysis
  • Related Work
  • Implementation
  • Future Work
slide-33
SLIDE 33

May 13, 2004 Otmar Hilliges 33/36

Future Work

  • Improve UI of development environment
  • Add convenience functionality

– Palettes – Toolbars – Repository of interaction atoms.

  • Programming by example
  • Authoring within Augmented Reality.
slide-34
SLIDE 34

May 13, 2004 Otmar Hilliges 34/36

Future Work II

  • Extensions to the DWARF UI architecture:

– User model. – Improved recognition techniques and multi-modal integration using Bayes nets and hidden Markov chains. – API for device integration.

slide-35
SLIDE 35

May 13, 2004 Otmar Hilliges 35/36

Outline

  • Introduction
  • Requirements Analysis
  • Related Work
  • Implementation
  • Future Work
slide-36
SLIDE 36

May 13, 2004 Otmar Hilliges 36/36

Questions

Any Questions ? Thank You!