Problem Statement Representing the Environment User Interaction - - PowerPoint PPT Presentation

COMPGV07 / M076 - Virtual Environments

• Systems

Simon Julier

Department of Computer Science, UCL S.Julier@cs.ucl.ac.uk http://www.cs.ucl.ac.uk/teaching/VE

Structure

• Problem Statement
• Representing the Environment
• User Interaction
• Execution Models
• XVR

2

Problem Statement

• Problem Statement
• Representing the Environment
• User Interaction
• Execution Models
• XVR

3

Reminder

User Interface Devices Environment User Synthetic Environment Real Environment Mediated Medium

4

Challenges in Developing VE Systems

• Use of lots of special (and often badly made) hardware
• Complex uni-modal and multi-modal interactions
• Fully 3D environment
• Complex manipulations of the environment
• Must run in real-time
• Scalable
• Distributed

5

Lack of Standards

• However, there are almost no universally agreed

software systems, standards and formats for VE

• Possible reasons:

– The field is still in its formative stages, and so we don’t know what the “right” answer is yet – There simply aren’t that many VE systems at the moment and nobody has thrown significant money at it – 2D isn’t that much better with many warring toolkits

6

A Virtual Reality Platform

Base Operating System

7

User Application Virtual Environment Platform

Inside the Platform

Graphics Rendering Audio Rendering Haptic Scene-Graph Network Master Environment Input Devices External Databases Graphics Scene-Graph Interaction Processing Audio Scene-Graph

8

Haptic Rendering User Application

Different Components Are Very Different

• Video (N copies – for stereo and multiple screens)

– Maintain copy of visual state – Render as fast as possible (~60Hz) – Synchronise with other renders

• Audio

– Maintain copy of audio state – Render without glitches (requires fast interrupt)

• Haptics

– Maintain copy of haptic data Render as fast as possible (~1000Hz)

9

External Databases

Responsiveness Means Spaghetti

Graphics Rendering Audio Rendering Haptic Rendering Haptic Scene-Graph Master Environment Input Devices Graphics Scene-Graph Interaction Processing Audio Scene-Graph User Application

10

Complicated, Coordinated System

11

Execution Model External Databases

The Execution Model

Graphics Rendering Audio Rendering Haptic Rendering Haptic Scene-Graph Master Environment Input Devices Graphics Scene-Graph Interaction Processing Audio Scene-Graph User Application

12

Network

Covered in This Lecture

Graphics Rendering Audio Rendering Haptic Scene-Graph Network Master Environment Input Devices External Databases Graphics Scene-Graph Interaction Processing Audio Scene-Graph

13

Haptic Rendering User Application Execution Model

Representing the Environment

• Problem Statement
• Representing the Environment
• User Interaction
• Execution Models
• XVR

14

Reminder: What is the Environment Made Of?

• Geometry defines the

space

• Contents is data
• Dynamics is code or

rules to change content

• The closest thing to a

standard is DIS

Contents: Actors and Objects Geometry: Dimensions, Metrics and Extent Dynamics: Interaction Rules

“Virtual Environments and Environmental Instruments”, S. Ellis, 1996

Environment

Motivation for DIS (SIMNET)

• Born out of needs for large-

scale military simulations:

– Hundreds of different types of entities – Dozens of servers scattered throughout the world – Real-time – Man-in-the-loop

• Complicated environments
• Complicated interactions

16

DIS Environmental Model

• World modelled as a set of entities (=objects)

– All entity locations available to all entities – All entities can serve as actors – All interactions between entities via events – Networking achieved using a mix of approaches:

• Ground truth information
• State change information
• Dead reckoning
• Entities and events are described by their Protocol Data Units

(PDUs)

To be discussed in the networking lecture

}

17

Representing Entities with DIS

IEEE Standards 1278.1-1995 & 1278.1a-1998

18

Representing the Environment with DIS

• Environments are considered to be object states which

aren’t associated with a specific entity

• Environmental states can come in several flavours:

– Gridded data (e.g., terrain) – Point objects (e.g., trees) – Linear objects (e.g., roads) – Area objects (e.g., bogs)

• Although this is a well-defined standard, only a few

military simulators use it

19

Reminder: What is the Environment Made Of?

Contents: Actors and Objects Geometry: Dimensions, Metrics and Extent Dynamics: Interaction Rules

“Virtual Environments and Environmental Instruments”, S. Ellis, 1996

Environment

Geometry

• Defines the “space” that the virtual environment is

embedded within

• It contains:

– Dimensionality: The degree of freedom of the position vector – Metric: The basic mathematical rules for defining order, distance, etc. – Extent: The range of possible values of the position vector

21

Describing Environment Geometry

• The environment geometry can be in many forms:

– Euclidean: simple (x, y, z); suitable for many applications

22

Cartesian Representations of Differing Extents

23

Describing Environment Geometry

• The environment geometry can be in many forms:

– Euclidean: simple (x, y, z); suitable for many applications – Locally linear: e.g., tangent planes for simulation over Earth’s surface – Nonlinear: can define arbitrary relationships between parts of environment

• An interesting example is a locale

24

Diamond Park

• Mitsubishi Research

Labs wanted to create a multi-user, collaborative environment in 1995

• Decentralised, scalable

interaction is key

• They solved this by

partitioning the world into a set of locales

25

Locales

• Environment is decomposed into a set of locales:

– Bounded regions of space – Regions can overlap one another – Transitions through portals

• Each object lies in a single locale
• Activities within one locale are not transmitted to
• bjects in another locale

26

Simple Locales: Disjoint Spaces

Locale 1 Locale 2 Locale 3

27

Complicated Locales: Overlapping Spaces

Locale 1 Locale 2 Locale 3

28

Contents

• The environment is populated by objects and actors
• Objects

– Discrete and identifiable – Described by property vectors

• Actors are objects that initiate interactions
• The self is a special kind of actor with a point-of-view

29

Describing Content Geometry

• Objects need to have a description in physical space
• Implicit or assumed to be 3D Cartesian coordinates with a 1m

unit scale usually

• Described in two steps:

– Describe the basic form of the environment

• 3D models, usually polygonal, there are standards for this

– Add properties to objects

• Visual properties: colour, texture, shading, …
• Sound properties: sources, reflectivity, …
• Material properties: weight, elasticity, …
• Semantic properties: (name, role, age, …)
• No standards for this
• Often implemented using a scene-graph

30

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Arbitrary graph

31

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Directed graph

32

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Directed acyclic graph

33

Graphs

• A graph consists of

vertices and edges

• Vertices define the “state”

information

• Edges define

“relationships”

• Scene-graphs are

directed and acyclic

Arbitrary graph Directed graph Directed acyclic graph

34

Scene-graphs

• In a scene-graph, vertices are
• ften called nodes

– Store state information – Can include arbitrary property information

• All graphs have a root node

which defines the base of the tree

• All other nodes divided into two

types:

– Group nodes – Leaf Nodes

Root node Group nodes Leaf nodes

35

Group Nodes

• Group nodes have multiple nodes as children

– Child nodes can be other group nodes or leaf nodes

• Applies common state information to multiple objects

– State information propagates down the graph

• Examples include:

– Transformations – Switch nodes – Effects

• Bump mapping, scribing, specular highlights

36

Examples (OpenSceneGraph)

Anisotropic Lighting Scribing Cartoon Bumpmapping

37

Leaf Nodes

• Leaf nodes cannot have children
• State information relates to the appearance of specific
• bjects
• Examples include:

– Geode

• Container of drawable objects such as shapes and text

– Imagery – Impostors

38

Examples (OpenSceneGraph)

Impostors Billboards

39

Example Haptic Scene-Graph

40

Dynamics

• These are the rules of interaction between the

contents

• These can be:

– Differential equations of Newtonian dynamics to describe kinematic and dynamic relationships – Grammatical rules for pattern-matched triggered actions

• Many different ways of doing this from imposing

numerical approximations to Newtonian physics, through to plain old C++ / Java / XVR coding

41

Implementing Dynamics as Standalone Processes

• Dynamics implemented

as separate processes / threads

• Can change state of the

graph in arbitrary ways

– Change values of nodes – Add / remove nodes

Dynamics

42

Examples of Dynamical Systems

• In DIS, entities can initiate environmental processes

– Persistent changes which occur in the environment even after the entity has “gone by” (e.g., contrails)

• In XVR, the PhysX physics engine can be used to

control the movement of objects

43

Implementing Dynamics Within the Scene-Graph

• Fairly “autonomous”

dynamics can be achieved by embedding dynamics within the scenegraph

• Animations are group nodes

which apply state changes to their children

• Examples include:

– Animation paths – Particle systems

Animation Node Animated nodes44

Example Animation and Particle System

45

Generalised Dynamics: Application Nodes

• Pre-defined behaviours allow lots of effects but are autonomous

– Script nodes (e.g., in VRML) can be used to generalise behaviour

• Most extreme example are application nodes:

– User application is written as a group node in the scenegraph – Application contains certain resources (e.g., viewport to display graphics) – Application owns and manages all of the nodes beneath it – Unifies application and environment state

• Capabilities include:

– Multiple applications in same environment – Load balancing – Dynamic workgroup management

46

Distributed Application Nodes

47

User Interaction

• Problem Statement
• Representing the Environment
• User Interaction
• Execution Models
• XVR

48

Managing Data from Input Devices

• So far we’ve talked

about the environment

• However, the user

actively participates in the environment as a type of actor

• The way the user

interfaces with the system is through the input devices

49

Complexity of Input Devices

• VR systems present unique challenges to the design
• f user interfaces:

– 6 degrees of freedom – Many types of interactions – Lots of different configurations of devices available – No agreed standards on what is the “right way” to navigate / interact with the environment

• One means of capturing the flexibility is to use a

dataflow model

50

Data Flow Model

• Processing consists of a series of filters
• Each filter has multiple input ports and a single output port
• Outputs from one filter can be treated as inputs to other filters
• Information sources are raw source of information (e.g.,

devices)

• Information sinks are final destination (e.g., applications)

Source 1 Filter Source 2 Port 1 Port 2 Output Filter Sink Source 3 OpenTracker

51

Example Hybrid System

• Combines 2D tracker (blue) with 3D vision-based tracker (black

square is fiducial marker)

52

Execution Models

• Problem Statement
• Representing the Environment
• User interaction
• Execution Models
• XVR

53

Execution Model Ties Everything Together

• So far we we’ve talked about a disparate set of

systems:

– A master environment – Separate representations for different output modes – User interfaces for controlling the environment

• The execution model “glues” all these parts together

– Closely related to distributed systems as well

54

Execution Models Ties Things Together

• Example:

– The position on an object is changed – The update needs to be reflected in:

• The master database
• The different scenegraphs
• Over the network (if connected)

– How can all of this be coordinated?

• Two main models:

– Kernel model – Actor/object model (events)

55

Simple Kernel Model

• Treats a VR application like a traditional graphical application:

while(true) { read_trackers(); set_body_position();//Changes scene graph do_animation(); //Changes scene graph render_left_eye(); render_right_eye(); render_sound(); poll_trackers();

}

do_animation

• After the user’s position is set by the read_trackers

and set_body_position (see later in course) …

• … do_animation manipulates the scene
• As we have noted, it is responsible for:

– Animations, physics, interaction

• This is where the standalone or all the within-scene-

graph dynamics are updated

Kernel: Application Runtime App: Frame Functions Draw Manager App: Initialization Gadgeteer: Device Update Kernel: System Reconfig

Kernel Model for VRJuggler

58

Problems with Simple Kernel Model

• Everything happens in serial order
• Rendering only happens at a fixed rate, so if part of

the animation slows down, the rendering slows down (very noticeable in many video games!)

• What if there are different output requirements such as

haptics (1000Hz)?

• What if the input rates are much higher (e.g. 200Hz)

Modified Kernel Model (1)

• Has a fast loop which calls different functions at different frame

rates:

while(true) { fast_function(); if (elapsed>30ms) slow_function(); } fast_function() {read_trackers(), do_haptics();} slow_function() {render_left_eye();render_right_eye(); render_sound();}

Modified Kernel Model (2)

• Uses a simple main loop, for the video rendering, but

“background” threads for reading devices

• Main thread is very much like the Simple Kernel Model, but the

function read_trackers and poll_trackers are now in a separate thread, and the main thread just copies information

• This is a good match to how operating systems actually

schedule non-blocking input and output

Pros / Cons of the Kernel Model

• Advantages:

– Simple to understand – Application programmer keeps their own data structures within the loop

• Disadvantages:

– Implementation needs care because of different update rates – Usually requires some awareness of parallel programming issues – Lots of complexity ends up in the do_animation() method

Actor Model

• Virtual environment is

realised by a set of collaborating asynchronous processes (actors)

• Actors send messages to
• ne another
• Processes share a

common database

• Database typically
• rganised around a scene-

graph

Database Audio Video1 Video2

Tracking

Speech

Collision Application

63

Actor Roles(1)

• Video (N copies – for stereo and multiple screens)

– Maintain copy of visual state – Render as fast as possible (~60Hz) – Synchronise with other renders

• Audio

– Maintain copy of visual state – Render without glitches (requires fast interrupt)

• Haptics

– Maintain copy of haptic data Render as fast as possible (~1000Hz)

64

Actor Roles(2)

• Tracking

– Read physical devices as fast as possibe (~100Hz) – Push data into scene-graph

• Collision

– Compute collisions based on geometry change

• Application

– Run simulation at necessary rate (1Hz-1000Hz) – Synchronise inconsistent state

65

Setting State on a Local Object

setXLocal() getX() X x; createSetXEvent() setX() sendEvent() Observers Object

66

Updating State from Remote Change

setXLocal() getX() X x; setX() Observers Object acceptEvent() executeSetXEvent()

67

Pros / Cons of the Actor Model

• Advantages:

– Application program does not care about distribution / what rendering systems used – Update rates and parallel processing issues handled by mutexes and buffering event objects – Complex chains of events can be implemented

• Disadvantages:

– Difficult to understand – Difficult to code – Can lead to strange cyclic dependency effects

68

XVR

• Problem Statement
• Representing the Environment
• User interaction
• Execution Models
• XVR

69

XVR

• Developed at PERCRO to support VE research

70

XVR Platform

71

Modified Kernel Model

72

Callback Structure

73

Multi-Wall Clustering

74

Sort-Late Network Driver

75

Summary

• Developing a VE system can be very challenging
• There are a lack of standards; lots of systems are “built by

hand”

• The Master Environment consists of geometry, content and

dynamics

• This modifies / is modified by several logically concurrent

processes (rendering, collision, audio etc…)

• Execution models need to reflect this concurrenc
• XVR uses a modified kernel model

76