Virtual Environments: System Architectures Anthony Steed Simon - - PowerPoint PPT Presentation

Virtual Environments: System Architectures

Anthony Steed Simon Julier

Department of Computer Science University College London http://www.cs.ucl.ac.uk/teaching/VE

Outline

• Problem Statement
• Representing the Environment
• User dynamics
• Execution Models

Problem Statement

• Problem Statement
• Representing the Environment
• User dynamics
• Execution Models

Reminder - VE is an Immersive, Mediated Communication Medium

User Interface Devices Environment User Synthetic Environment Real Environment Mediated Medium

Key Requirements of Systems

• Speed of update

– Especially in rendering and haptics

• Latency

– Time from tracker update to display change should be as short as possible (ideally <75ms)

• Consistency

– Environment state should be consistent with input

• Expressivenees

– Environment should respond to a range of user input

Modules and Responsibilities

Graphics Rendering Audio Rendering Haptic Rendering Haptic Scene-Graph Network Master Environment Input Devices External Databases Graphics Scene-Graph Interaction Processing Audio Scene-Graph

Different Display Modes Have Different Requirements

• Video (N copies – for stereo and multiple screens)

– Maintain copy of visual state – Render as fast as possible (~60Hz) – Synchronise with other renders

• Audio

– Maintain copy of audio state – Render without glitches (requires fast interrupt)

• Haptics

– Maintain copy of haptic data Render as fast as possible (~1000Hz)

Representing the Environment

• Problem Statement
• Representing the Environment
• Dynamics
• Execution Models

Environment

• Environment is a broad term, but what is it we are

actually modelling?

– Something that can be rendered and interacted with such that

• We utilize capabilities of display system
• Maximize the opportunity for interaction
• Ellis states that VEs have 3 main components:

– Content – Geometry – Dynamics

Contents

• Environment is made up of discrete items known as
• bjects and actors
• Objects

– Discrete and identifiable – Described by property vectors

• Actors are objects that initiate interactions
• The self is a special kind of actor with a point-of-view

Representing the Contents

• Unfortunately little agreement about conventions,

schema, specifications for describing environments

– Relates to issues about defining and using ontologies – Standards where they exist usually focus on visual representation – Possibility that some standards will emerge

• Well-known example is the Distributed Interactive

Simulation (DIS) Entity Model

Motivation for DIS (SIMNET)

• Born out of needs for large-

scale military simulations:

– Hundreds of different types of entities – Dozens of servers scattered throughout the world – Real-time – Man-in-the-loop

• Complicated environments
• Complicated interactions

DIS Environmental Model

• World modelled as a set of entities

– All entity locations available to all entities – All entities can serve as actors – All interactions between entities via events – Networking achieved using a mix of approaches:

• Ground truth information
• State change information
• Dead reckoning
• Entities and events are described by their Protocol Data Units

(PDUs)

To be discussed in Week ~11

}

Representing Entities with DIS

IEEE Standards 1278.1-1995 & 1278.1a-1998

Representing the Environment with DIS

• Environments are considered to be object states which

aren’t associated with a specific entity

• Environmental states can come in several flavours:

– Gridded data (e.g., terrain) – Point objects (e.g., trees) – Linear objects (e.g., roads) – Area objects (e.g., bogs)

Geometry

• Description of the environmental field of action
• Contains:

– Dimensionality: The degree of freedom of the position vector – Metric: The basic mathematical rules for defining order, distance, etc. – Extent: The range of possible values of the position vector

• Defines the “space” where the environment is

described

Describing Environment Geometry

• Typically Euclidean

– simple (x, y, z); suitable for many applications

• However not that straight forward

– Euclidean not that useful for describing geometry on spheroids (e.g. planets)

• Use a locally linear model (i.e. on a tangent plane)

– Not terribly useful in large-scale collaborative environments (everyone wants to be at (0,0,0))

• Use a differential geometry model (i.e. everyone sets their own

coordinates and connections between models have relative transforms)

Describing Object Geometry

• Objects need to have a description in physical space
• Implicit or assumed to be 3D Cartesian coordinates with a 1m

unit scale usually

• Described in two steps:

– Describe the basic form of the environment

• 3D models, usually polygonal, there are standards for this (VRML, DXF, OBJ)

– Add properties to objects

• Visual properties: colour, texture, shading, …
• Sound properties: sources, reflectivity, …
• Material properties: weight, elasticity, …
• Semantic properties: (name, role, age, …)
• No standards for this
• Often implemented using a scene-graph

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Arbitrary graph

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Directed graph

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Directed acyclic graph

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Arbitrary graph Directed graph Directed acyclic graph

Scene-graphs

• In a scene-graph, vertices are
• ften called nodes

– Store state information – Can include arbitrary property information

• All graphs have a root node

which defines the base of the tree

• All other nodes divided into two

types:

– Group nodes – Leaf Nodes

Root node Group nodes Leaf nodes

Group Nodes

• Group nodes have multiple nodes as children

– Child nodes can be other group nodes or leaf nodes

• Applies common state information to multiple objects

– State information propagates down the graph

• Examples include:

– Transformations – Switch nodes – Effects

• Bump mapping, scribing, specular highlights

Examples (OpenSceneGraph)

Anisotropic Lighting Scribing Cartoon Bumpmapping

Leaf Nodes

• Leaf nodes cannot have children
• State information relates to the appearance of specific
• bjects
• Examples include:

– Geometry – Image based rendering

• Billboards
• Impostors

Examples (OpenSceneGraph)

Impostors Billboards

Dynamics

• These are the rules of interaction between the

contents

• These can be:

– Differential equations of Newtonian dynamics to describe kinematic and dynamic relationships – Grammatical rules for pattern-matched triggered actions

• Many different ways of doing this from imposing

numerical approximations to Newtonian physics, through to plain old C++ / Java / XVR coding

Implementing Dynamics as Standalone Processes

• Dynamics implemented

as separate processes / threads

• Can change state of the

graph in arbitrary ways

– Change values of nodes – Add / remove nodes

Dynamics

Implementing Dynamics Within the Scene-Graph

• Fairly “autonomous”

dynamics can be achieved by embedding dynamics within the scenegraph

• Animations are group nodes

which apply state changes to their children

• Examples include:

– Animation paths – Particle systems

Animation Node Animated nodes

Example Animation and Particle System

Generalised Dynamics: Application Nodes

• Pre-defined behaviours allow lots of effects but are autonomous

– Script nodes (e.g., in VRML) can be used to generalise behaviour

• Most extreme example are application nodes:

– Entire VR application is written as a group node in the scenegraph – Application contains certain resources (e.g., viewport to display graphics) – Application owns and manages all of the nodes beneath it – Unifies application and environment state

• Capabilities include:

– Multiple applications in same environment – Load balancing – Dynamic workgroup management

User Dynamics

• Problem Statement
• Representing the Environment
• User dynamics
• Execution Models

Managing Data from Input Devices

• So far we’ve talked

about objects and actors

• However, the user

actively participates in the environment as a type of actor

• The way the user

interfaces with the system is through the input devices

Complexity of Input Devices

• VR systems present unique challenges to the design
• f user interfaces:

– 6 degrees of freedom – Many types of interactions – Lots of different configurations of devices available – No agreed standards on what is the “right way” to navigate / interact with the environment

• One means of capturing the flexibility is to use a

dataflow model

Data Flow Model

• Processing consists of a series of filters
• Each filter has multiple input ports and a single output port
• Outputs from one filter can be treated as inputs to other filters
• Information sources are raw source of information (e.g.,

devices)

• Information sinks are final destination (e.g., applications)

Source 1 Filter Source 2 Port 1 Port 2 Output Filter Sink Source 3 OpenTracker

Example Hybrid System

• Combines 2D tracker (blue) with 3D vision-based tracker (black

square is fiducial marker)

Execution Models

• Problem Statement
• Representing the Environment
• User dynamics
• Execution Models

Execution Model Ties Everything Together

• So far we we’ve talked about a disparate set of

systems:

– A master environment – Separate representations for different output modes – User interfaces for controlling the environment

• The execution model “glues” all these parts together

– Closely related to distributed systems as well

Execution Models Tying Things Together

• Example:

– The position on an object is changed – The update needs to be reflected in:

• The master database
• The different scenegraphs
• Over the network (if connected)

– How can all of this be coordinated?

• Two main models:

– Kernel model – Actor/object model (events)

Simplified Kernel Model

• Treats a VR application like a traditional graphical application:
• In practice, it’s never as simple as this…

wh i l e ( t r ue ) { r ead_ t r acke r s ( ) ; se t_body_pos i t i

• n

( ) ; do_an i mat ion( ) ; r ende r _ le f t _eye ( ) ; r e nde r_ r i gh t_eye ( ) ; r ende r _sound ( ) ; po l l _ t r acke rs( ) ;

}

Kernel: Application Runtime App: Frame Functions Draw Manager App: Initialization Gadgeteer: Device Update Kernel: System Reconfig

Kernel Model for VRJuggler

Pros / Cons of the Kernel Model

• Advantages:

– Simple to understand – Application programmer keeps their own data structures (no need for a scene-graph)

• Disadvantages:

– Implementation needs care because of different update rates – Usually requires some awareness of parallel programming issues – Lots of complexity ends up in the do_an i mat ion( )method

• XVR addresses some of these issues through its threading /

event model

XVR Threading Model

Actor Model

• Virtual environment is

realised by a set of collaborating asynchronous processes (actors)

• Actors send messages to
• ne another
• Processes share a

common database

• Database typically
• rganised around a scene-

graph

Database Audio Video1 Video2

Tracking

Speech

Collision Application

Setting Object State Using in the Actor Model

• Setting the object state is often achieved using the

subject-observer design pattern

• The object in the database is the subject
• Different renderers / networking systems are the
• bservers
• When the subject’s state is updated, the observers are

automatically notified

Pros / Cons of the Actor Model

• Advantages:

– Application program does not care about distribution / what rendering systems used – Update rates and parallel processing issues handled by mutexes and buffering event objects – Complex chains of events can be implemented

• Disadvantages:

– Difficult to understand – Difficult to code – Can lead to strange cyclic dependency effects

Summary

• Representing the environment is difficult

– The representation has to be rich enough to capture the contents, geometry and dynamics – Each display mode requires its own form of the environment to optimise the display

• Want to make content as rich as possible to support dynamic

models

– Otherwise behaviour is expressed only in code.

• At run-time there are logically concurrent processes (rendering,

collision, audio etc…)

• Execution models need to reflect this concurrency