Lindemans Lectures: Virtual Reality & Serious Games (Part 1) - - PowerPoint PPT Presentation

Lindeman’s Lectures: Virtual Reality & Serious Games

(Part 1) Robert W. Lindeman

Assistant Professor Interactive Media & Game Development Human Interaction in Virtual Environments (HIVE) Lab Department of Computer Science Worcester Polytechnic Institute gogo@wpi.edu

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 2

Five-Lecture Structure

 July 15

 Introduction to Game Development

 July 16

 Game Design (part 1)

 July 23

 Game Design (part 2)

 July 29

 Virtual Reality / Serious Games

 July 30

 Future Gaming (Natural Interaction, MMOs, Mobile)

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 3

Today’s Outline

What is Virtual Reality? Why is it important? What are Serious Games?

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 4

Motivation

 Much excitement (and hype) about how VR was

going to change things

 VR has not made inroads into everyday life

Lagging technology Lack of understanding of usability issues Lack of "killer app"

 Still remains mainly in research labs  Video games show great promise  Training scenarios - surgery, military, therapy

 Long-Term Goal

 Make VR more usable

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 5

What is Virtual Reality?

You tell me!

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 6

Virtual Reality Systems



1929 – Link Flight Simulator



1946 – First computer (ENIAC)



1956 – Sensorama



1960 – Heileg’s HMD



1965-68 – The Ultimate Display



1972 – Pong



1973 – Evans & Sutherland Computer Corp.



1976 – Videoplace



1977 – Apple, Commodore, and Radio Shack PCs



1979 – First Data Glove [Sayre] (powerglove -89)



1981 – SGI founded



1985 – NASA AMES



1986-89 – Super Cockpit Program



1990s – Boom Displays



1992 – CAVE (at SIGGRAPH)



1995 – Workbench



1998 – Walking Experiment

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 7

Link Flight Simulator

 1929 - Edward Link

develops a mechanical flight simulator

 Train in a synthetic

environment

 Used mechanical linkages  Instrument (blind) flying  http://www.wpafb.af.mil/

museum/early_years/ey1 9a.htm

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 8

Sensorama

Morton Heilig, 1956

Motorcycle simulator - all senses

• visual (city scenes)
• sound (engine, city sounds)
• vibration (engine)
• smell (exhaust, food)

Extend the notion of a ‘movie’

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 9

Sensorama by Morton Heilig (1960)

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 10

Heilig's HMD (1960)

Simulation Mask from Heilig’s 1960 patent

 3D photographic slides  WFOV optics with focus

control

 Stereo sound  Smell

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 11

Ivan Sutherland

The Ultimate Display (FIPS 1965)

 Data Visualization: “A display connected to a

digital computer…is a looking glass into a mathematical wonderland.”

 Body Tracking: “The computer can easily

sense the positions of almost any of our body muscles.”

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 12

Ultimate Display (cont.)

 Virtual Environments that mimic real environments:

“A chair display in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal.”

 VEs that go beyond reality: “There is no reason why

the objects displayed by a computer have to follow

• rdinary rules of physical reality with which we are

familiar.”

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 13

First HMD-Based VR

1965 - The Ultimate Display paper by Sutherland 1968 - Ian Sutherland’s HMD

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 14

Molecular Docking Simulator

 Incorporated force

feedback

 Visualize an abstract

simulation

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 15

Data Gloves

 Light, electrical or metal

detectors compute “bend”

 Electrical sensors detect pinches.  Force feedback mechanical

linkages

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 16

1985 - NASA Ames HMD

 McGreevy and and

Humphries

 Wearable immersive

HMDs

 LCD “Watchman”

displays

 LEEP Optics

 Led to VIVID, led by

Scott Fisher

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 17

FakeSpace Boom Display: Early 1990s

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 18

CAVE - 1992

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 19

Virtual Workbench-1995

(Responsive Workbench, Immersidesk, etc.)

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 20

Current Best VE

 UNC Pit Experiment  Fear of Heights a Strong

Response

 Thousands of visitors  Compelling Experience

 Haptics  Low Latency  High Visual Quality

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 21

VPL Founded - 1985

First VR Company VPL Research by Jaron

Lanier and Thomas Zimmerman

 Data Glove  Term: Virtual Reality

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 22

1995 - Effectiveness of computer-generated (VR) graded exposure in the treatment of acrophobia in American Journal of Psychiatry

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 23

Major Reinvigoration: Hardware Evolution

 High expense  PC performance surpasses Graphics

supercomputers

 SGI RealityEngine (300k tris – 1993)  XBox (150 mil tri/sec - 2001)  XBox360 (500 mil tri/sec - 2005)  Wii-mote

 Large Volume Displays  VR Estimated $3.4 billion industry in 2005

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 24

Background

 VR defined:

 Fooling the senses into believing they are

experiencing something they are not actually experiencing

 Virtual reality systems consist of:

 Graphical/audio/haptic/... rendering  Content  Tracking of people and objects  Collision detection  Interaction techniques  Optional, but common:

 Networking  Autonomous agents

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 25

Keys to Success

 High fidelity (or realism)

 Graphics, audio, haptics, behaviors, etc.

 Low latency

 Tracking  Collision detection  Rendering  Networking

 Ease of use

 Low cumber for users  Easy integration for programmers

 Compelling Content

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 26

The Senses

 See (Visual Sense):

 Visuals are excellent!

 Hear (Aural Sense):

 Spatialized audio is very good!

 Smell (Olfactory Sense):

 Very hard! Too many types of receptors.

 Touch (Haptic Sense):

 Application specific and cumbersome

 Taste (Gustatory Sense):

 We know the base tastes, but that is it!

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 27

See: Head-Mounted Displays

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 28

See: Projection-Based Environments

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 29

See: Projection-Based Environments (cont.)

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 30

Hear: Sound in VR

 Display techniques

 Multi-speaker output (sound cube)  Headphones  Bone-Conduction

 Waveform filtering

 Simple balance & volume control  Head-Related Transfer Functions

 Software "Standards"

 OpenAL  A3D from Aureal (RIP!)  VRSonic.com

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 31

Smell: Olfactory Sense

Two main problems

 Scent generation

 Tens of thousands of receptor types

 Scent delivery

 Easier problem

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 32

Smell: Air Cannon (Yanagida, 2004)

CLIP

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 33

Touch: Haptic Feedback in VR

 Tactile: Surface

properties

 Most densely populated

area is the fingertip (okay, it's the tongue)

 Kinesthetic: Muscles,

Tendons, etc.

 Also known as

proprioception

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 34

Vibrotactile Feedback Projects

TactaBoard and TactaVest

CLIP

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 35

Empirical Studies

TactaVest experiments

 Exposure during room clearing tasks  Spatial awareness  Team member location for team training  Robot tele-operation

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 36

Exposure Experiments

 Looking at the use of

spatialized vibrotactile feedback as a training aid

• n "victim" search tasks

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 37

Exposure Experiments (cont.)

 Use vibration to convey exposure  Results in ACM CHI 2005

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 38

Issues to be Addressed

Transfer effects from virtual to real

environments

 How do subjects perform after training in

VR?

Psychophysical issues

 Sensory substitution

Cognitive Issues

 Does the addition of haptic cues increase

cognitive load?

Multi-modal integration

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 39

Interaction in VR

Use of a keyboard and mouse is not

tractable

 Can't see them  Want to move around  No good 3D mappings

How can we allow easy interaction that

takes advantage of real-world experience?

 This is the problem that we need to solve!

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 40

Basic Interaction Tasks in VR (Bowman et al.)

 Object Selection

 What do I want to manipulate?

 Object Manipulation

 How can I manipulate it?

 Navigation

 Wayfinding: How do I know where I am, and

how to get where I am going?

 Travel: How do I get there? (locomotion)

 System Control

 How do I change system parameters?

 Symbolic Input

 Inputting text and numbers

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 41

Oh, I forgot One (Lindeman)

Killing

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 42

Dealing with Objects

 Problems

 Ambiguity  Distance

 Selection Approaches

 Direct / enhanced grabbing  Ray-casting techniques  Image-plane techniques

 Manipulation Approaches

 Direct position / orientation control  Worlds in miniature  Skewers  Surrogates Courtesy: D. Bowman

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 43

Navigation: Wayfinding

People get lost/disoriented easily Traditional tools

 Maps (North-up vs.

Forward-up)

 Landmarks  Spoken directions

Non-traditional

 Callouts  Zooming

Images: http://vehand.engr.ucf.edu/handbook/Chapters/Chapter28/Chapter28.html

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 44

Navigation: Travel

Problems

 Limited physical space, unlimited virtual

space

 Cables

Approaches

 Fly where you point/look  Treadmills  Walking in place  Big track ball

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 45

Overview

Travel

 Getting from one place to another

Wayfinding

 Means knowing

 Your current location (here)  The location of your destination (there)  A (partial) route for getting there from here

These are related, but are really two

large separate problems

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 46

Travel: Key Research Problems

 Limited physical space, possibly infinite virtual

space

 Think Holodeck

 Different types of travel

 Walking, running, turning, side stepping, back

stepping, crawling, quick start/stop, ...

 Need to do other things while traveling

 Usually, travel is not the goal of your current task

 It is very easy to get (cognitively) lost in virtual

reality

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 47

Support for General Walking

Multi-sensory cues

 Visual  Auditory  Tactile  Kinesthetic  Vestibular  Cognitive

Each technique used for travel has more

• r less support for each of these

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 48

Overview of Travel Approaches

Gestural

 Hand  Head  Foot (walking in place)  Body (real walking, re-directed walking)

Device

 Hand-held devices (joystick, gamepad, 2D

mouse)

 Platforms

 Passive (tilt, pressure, VirtuSphere)  Active (treadmills, steppers, CirculaFloor)

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 49

Gestures for Travel

Hand typically... Head...

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 50

Gestures (cont.)

Walking in place (Gaiter [Templeman])

 Forward/backward/side-step gestures  Go prone, run, small real steps

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 51

Gestures (cont.)

Redirected walking (UNC-CH)

(movie)

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 52

Devices

Hand-held

 Mouse, joystick, gamepad, Wiimote, etc.

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 53

Platforms

Passive

 Tilt boards  Wii Fit

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 54

Platforms (cont.)

VirtuSphere

CLIP

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 55

Platforms (cont.)

Virtual Perambulator (Iwata 1996)

CLIP

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 56

Platforms (cont.)

Torus Treadmill

(Iwata 1999)

CLIP

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 57

Platforms (cont.)

GaitMaster

(Iwata 2000)

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 58

Platforms (cont.)

Powered Shoes

(Iwata 2006)

CLIP

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 59

Platforms (cont.)

String Walker

(Iwata 2007)

CLIP

R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development 60

Platforms (cont.)

CirculaFloor

(Iwata 2004)

CLIP