COMPM076 / GV07 - Introduction to Virtual Environments: Mixed - - PowerPoint PPT Presentation

▶
compm076 gv07 introduction to virtual environments mixed
SMART_READER_LITE
LIVE PREVIEW

COMPM076 / GV07 - Introduction to Virtual Environments: Mixed - - PowerPoint PPT Presentation

Sep 30, 2022 199 likes •1.31k views

COMPM076 / GV07 - Introduction to Virtual Environments: Mixed Reality Simon Julier Department of Computer Science University College London http://www.cs.ucl.ac.uk/teaching/VE Structure Introduction Display Systems Tracking

slide-1
SLIDE 1

COMPM076 / GV07 - Introduction to Virtual Environments: Mixed Reality

Simon Julier Department of Computer Science University College London http://www.cs.ucl.ac.uk/teaching/VE

slide-2
SLIDE 2

Structure

  • Introduction
  • Display Systems
  • Tracking Systems
  • Display Techniques
  • Interaction Methods

2

slide-3
SLIDE 3

Introduction

  • Introduction
  • Display Systems
  • Tracking Systems
  • Display Techniques
  • Interaction Methods

3

slide-4
SLIDE 4

Virtual Environments

  • The focus of this module so far has been on

developing virtual environments

– The user is surrounded by an entirely virtual world – Objectives are to do things like induce immersion or presence

  • However,VE is at just one end of the mixed reality

continuum

4

slide-5
SLIDE 5

The Mixed Reality Continuum

5

slide-6
SLIDE 6

What is Mixed Reality?

  • MR systems combine virtual and real objects

6

slide-7
SLIDE 7

But This Includes Film Special Effects…

I, Robot

7

slide-8
SLIDE 8

What is Mixed Reality?

  • MR systems combine virtual and real objects
  • They run interactively and in real time

8

slide-9
SLIDE 9

But This Includes Heads Up Displays

SportVUE MC1 Wireless Heads Up Display

9

slide-10
SLIDE 10

What is Mixed Reality?

  • MR systems combine virtual and real objects
  • They run interactively and in real time
  • They register (align) virtual and real objects with
  • ne another

10

slide-11
SLIDE 11

The Ultimate MR System

“Augmented Reality, a New Way of Seeing”, S. K. Feiner, Scientific American, April 2002

11

slide-12
SLIDE 12

World-Fixed vs. Camera-Fixed MR

  • When we register with the environment, we have

to know where the real world objects are

  • Two types of MR system can be developed:

– World fixed – Camera fixed

12

slide-13
SLIDE 13

World-Fixed MR Application

13

slide-14
SLIDE 14

Heads Up Display

14

slide-15
SLIDE 15

Broadcasting

racef/x by sportvision

15

slide-16
SLIDE 16

Broadcasting

1st & ten by sportvision

16

slide-17
SLIDE 17

Surveying

Courtesy of Mark Francis, Topcon

17

slide-18
SLIDE 18

Surveying

Topcon GPT-7000i Wide field of view camera Narrow field of view camera Captured image + overlays

18

slide-19
SLIDE 19

Surveying

19

slide-20
SLIDE 20

Pre-visualisation

20

slide-21
SLIDE 21

Synthetic Vision for UAVs

slide-22
SLIDE 22

Navigation

22

slide-23
SLIDE 23

Navigation

Ruggedised pan tilt zoom camera Annotated video shown on bridge

23

slide-24
SLIDE 24

Navigation

24

slide-25
SLIDE 25

Advertising

25

slide-26
SLIDE 26

Advertising

26

slide-27
SLIDE 27

Location-Aware Information

Layar

27

slide-28
SLIDE 28

Camera-Fixed MR Application

28

slide-29
SLIDE 29

Lego Digital Box

29

slide-30
SLIDE 30

MR For Postage

US Postal Service Virtual Box Simulator Demo

30

slide-31
SLIDE 31

Mixed Reality and Games Consoles

“Eye of Judgement”, Sony Computer Entertainment

31

slide-32
SLIDE 32

Mobile AR Systems

32

Interface Interaction Display Tracking

slide-33
SLIDE 33

Challenges Posed by MR

  • In some ways, MR is easier:

– Because it is anchored to the real world, tracking and navigation metaphors come for free

  • However, because we are anchoring the graphics

to the real world, other challenges arise:

– The type of display critically affects what can be shown – The accuracy of the tracking system critically affects how well the real and virtual objects are registered – The way in which the graphics interacts with the real world controls depth perception, etc.

33

slide-34
SLIDE 34

Summary

  • Mixed reality is a generalisation of VEs to

incorporate both real and virtual objects

  • Applications can be world-fixed or camera-fixed
  • They are very widely used, even today
  • The challenges for building good MR systems

relate with how the virtual and real content interact

34

slide-35
SLIDE 35

Display Systems

  • Introduction
  • Display Systems
  • Tracking Systems
  • Display Techniques
  • Interaction Methods

35

slide-36
SLIDE 36

Building an AR System

36

Display

slide-37
SLIDE 37

Display Systems

  • There are several kinds of display technology

which can be used which depend upon the application:

– Video see-through displays – Optical see-through displays – Projection displays

slide-38
SLIDE 38

Video See-Through Display

  • The scene is observed by a camera
  • The graphics are overlaid on the video stream

from the camera

38

slide-39
SLIDE 39

Video See-Through HMDs

39

slide-40
SLIDE 40

Mobile Phones

  • Given the increase in memory and computational

capability, together with handy built in cameras, mobile phones are now a major MR platform

40

slide-41
SLIDE 41

Tablet-Based PC

VisTracker and AR Demo courtesy of InterSense

41

slide-42
SLIDE 42

Any Content Can be Inserted

42

NRL Virtual Combat Trainer

slide-43
SLIDE 43

For Example…

43

UNC Needle-Guided Biopsy Guided maintenance of equipment

slide-44
SLIDE 44

Disadvantages of Video See Through Display

  • Can only see part of real world collected by the

camera

– Narrow field of view – Low contrast

  • If the power fails, you can’t see anything!
  • Focal length of device is fixed to distance of

display

– Includes both real world and virtual objects at any depth

44

slide-45
SLIDE 45

Optical See-Through Display

  • Considered the “gold standard”
  • A user wears a see-through head mounted display
  • The real world and graphics are merged using

some kind of optical combiner

45

slide-46
SLIDE 46

Examples of See-Through HMDs

46

slide-47
SLIDE 47

Advantages of Optical See-Through Displays

  • The real world can be seen all the time

– Wide field of view – High contrast – High colour resolution

  • If the display loses power, you can still see the

real world

47

slide-48
SLIDE 48

Optical See-Through Display

48

slide-49
SLIDE 49

Optical See-Through Display

49

slide-50
SLIDE 50

Graphics Can’t Block “Real World”

50

slide-51
SLIDE 51

Light Leakage is a Significant Problem

51

slide-52
SLIDE 52

High-Powered Displays Partial Solution

52

slide-53
SLIDE 53

Projective Displays

  • User wears projector on head that casts images
  • nto scene
  • Reflection is visible

53

slide-54
SLIDE 54

Projective HMD in Action

54

slide-55
SLIDE 55

Advantages of Projective Displays

  • Graphics overlaid directly, in

terms of depth, with real world

  • The tape can be used to
  • cclude otherwise disruptive
  • bjects in the environment

55

slide-56
SLIDE 56

Disadvantages of Projective Displays

  • If power goes, everything goes
  • Retroreflective tape has to be put everywhere you

expect to see virtual content

  • Since intensity falls with the square of distance,
  • nly nearby objects can be seen brightly

56

slide-57
SLIDE 57

Projective MR

“EasyWeb” Show Reel

57

slide-58
SLIDE 58

Display Technology Summary

  • Three main classes of displays can be used:

– Video see-through – Optical see-through – Projective

  • All three have advantages and disadvantges
  • Video see-through is probably the most widely

researched and used:

– Simple – Can ‘overwrite’ any object in the scene

58

slide-59
SLIDE 59

Tracking Systems

  • Introduction
  • Display Systems
  • Tracking Systems
  • Display Techniques
  • Interaction Methods

59

slide-60
SLIDE 60

ARToolKit

  • The most widely used

system

  • Markers are 2D

rectangles with distinctive patterns

  • The pose of the

pattern is computed relative to the camera

60

slide-61
SLIDE 61

Tracking Systems

  • To show the information to the user, we have to

know where they are and where they are looking

  • You might think we can get away with just a GPS

and compass

– GPS gives position – Compass gives orientation

  • However, the performance can be pretty poor

61

slide-62
SLIDE 62

GPS Causes Latency

  • MARA, Nokia – Nokia 6680 with add-on box containing GPS

62

slide-63
SLIDE 63

Metal Causes Heading Errors

NRL Battlefield Augmented Reality System

63

slide-64
SLIDE 64

InterSense VisTracker

  • Hybrid vision-inertial

system

  • Camera looks for special

markers at low speed

  • Inertial system measures

velocity and acceleration at high speed

  • Two are combined to give

fast, accurate tracking

64

InterSense VisTracker

slide-65
SLIDE 65

Natural Marker-Based Systems

  • Using artificial markers is a pain

– You have to physically install them on site – You have to measure them to work out where they are – You have to make sure that you put enough of them in that the tracker can see enough to compute the pose

  • An alternative is to the unmodified environment

– Model-based – Natural marker-based

65

slide-66
SLIDE 66

Model-Based Tracking

  • The system already knows the structure of all the

key features in the environment it can track from

  • The system therefore detects for model features in

the frame

  • These are associated with the model
  • The camera pose is computed

66

slide-67
SLIDE 67

Hybrid Model-Based Tracking System

“Going Out”, G. Reitmayr and T. Drummond, Cambridge University MEMS IMU Camera Interfacing and power!

67

slide-68
SLIDE 68

Hybrid Model-Based Tracking System

68

“Going Out”, Gerhard Reitmayr and Tom Drummond, Cambridge University

slide-69
SLIDE 69

SLAM-Based Tracking

  • Developing models of the environment can be at

least as difficult as using artificial markers

  • Therefore, an alternative technique is to allow the

system to detect and build a map of its own environment

  • This approach is known as Simultaneous

Localisation and Mapping (SLAM)

69

slide-70
SLIDE 70

AR in Small Workspaces

70

slide-71
SLIDE 71

Parallel Tracking and Mapping

“Parallel Tracking and Mapping”, courtesy of Georg Klein and David Murray, University of Oxford

slide-72
SLIDE 72

PTAM on an iPhone

72

Courtesy of Georg Klein and David Murray, University of Oxford

slide-73
SLIDE 73

AR In Large Workspaces

“Mapping Large Loops with a Single Hand-Held Camera”, L. Clemente and A. Davison and I. Reid and J. Neira and J. Tardos

73

slide-74
SLIDE 74

Tracking System Summary

  • Reliable and accurate tracking is important
  • Wide-area tracking systems aren’t good enough
  • Hybrid and computer vision-based systems are in

the works

  • Global coverage is not here yet

74

slide-75
SLIDE 75

Display Techniques

  • Introduction
  • Display Systems
  • Tracking Systems
  • Display Techniques
  • Interaction Methods

75

slide-76
SLIDE 76

Building an AR System

76

Interface

slide-77
SLIDE 77

User Interfaces

  • To be of any use, the user has to understand the

information which is being shown to them

  • However, there are several issues we have to

grapple with, including:

– Environmental effects – Density of information – Occlusions – Unambiguous labelling

77

slide-78
SLIDE 78

Annotation and Labelling

  • One common issue is that we want to label the

environment

  • However, simple strategies like putting labels at

centroids isn’t good enough – need to look at the “screen space” of objects

78

slide-79
SLIDE 79

Labelling The Environment

  • One approach is to use view management

– Keep track of where the 3D objects project onto the screen – 2D annotations are laid out so that:

  • They fit in or near the space of the object
  • They do not interfere with one another
  • The temporal placement of annotations is

controlled using a finite state machine

slide-80
SLIDE 80

Labelling the Environment

Work out projection of objects onto display

slide-81
SLIDE 81

Labelling the Environment

Parameterise unoccupied space

slide-82
SLIDE 82

Labelling the Environment

  • The label location strategy is

a finite state machine

slide-83
SLIDE 83

Labelling the Environment

Place annotations into free areas

slide-84
SLIDE 84

Adaptive Label Management

slide-85
SLIDE 85

Depth Estimation and Depth Ordering

  • We often want to know the order or range of

virtual objects relative to the environment

  • Stereoscopic cues are of limited use
  • Secondary cues tend to bias virtual object further

away than real objects

  • Various supplementary cues can be provided to

convey depth information

slide-86
SLIDE 86

Viewing Occluded Objects

Threat location User Threat Location

86

slide-87
SLIDE 87

“X-Ray” Vision Through Display Styles

87

slide-88
SLIDE 88

Before

88

slide-89
SLIDE 89

After

89

slide-90
SLIDE 90

Developed Adaptation to Ambient Conditions

  • Color of text is function of

background over which it is drawn

– Algorithms identified by evaluation study at VaTech

  • Therefore, the colour of the

augmentation has to change based on scene composition

– Camera views scene – Color and intensity processed to change label colors Camera

slide-91
SLIDE 91

Developed Adaptation to Ambient Conditions

Label (“Parking Lot”) changes color to contrast with background

slide-92
SLIDE 92

Registration Errors Can Be Confusing

92

slide-93
SLIDE 93

Reduce Effects by Adapting the Display

Increasing registration error

93

slide-94
SLIDE 94

Bounding Regions for Objects

  • For each vertex in object:

– Project each vertex through projection equations to work out pixel position – Calculate mean and 1σ covariance ellipse due to registration errors – Approximate ellipses by set of points – Calculate bounding regions defined by these ellipses

World Head p p’ Screen y(k)= [ŷ(k), Y(k)] P xi(k)=[xi(k), Xi(k)] ^ M(k) = f(M1(x1(k)), … ,Mt(xt(k)))

slide-95
SLIDE 95

Registration Error Adaptation

Target designated

Target window

95

slide-96
SLIDE 96

Registration Error Adaptation

Target registration error region overlaps distracters

Target window Error bounds

96

slide-97
SLIDE 97

Registration Error Adaptation

Calculate overlap of target and distracter error regions

97

slide-98
SLIDE 98

Registration Error Adaptation

Add aggregate representation and text message

targetObject is in the middle 98

slide-99
SLIDE 99

Dealing With Information Overload

99

slide-100
SLIDE 100

Lots of Data is Extremely Confusing

100

slide-101
SLIDE 101

Results

101

slide-102
SLIDE 102

Interaction Methods

  • Introduction
  • Display Systems
  • Tracking Systems
  • Display Techniques
  • Interaction Methods

102

slide-103
SLIDE 103

Building an AR System

103

Interface Interaction

slide-104
SLIDE 104

Interaction

104

slide-105
SLIDE 105

Heads Up Interaction

What’s that object?

105

slide-106
SLIDE 106

Selection and Interaction

106

slide-107
SLIDE 107

Multimodal Integrator Architecture

Speech Recogniser Gesture Recogniser Multimodal Integrator Weighted list

  • f sentences

Weighted list

  • f selections

Object with highest probability

107

slide-108
SLIDE 108

Example Multi-modal Interaction

108

“Probabilistic Algorithms, Integration, and Empirical Evaluation for Disambiguating Multiple Selections in Frustum-Based Pointing”, G. Schmidt et al.

slide-109
SLIDE 109

Summary

  • Many interaction challenges are the same as

those in VE and the same techniques can be used

  • Multi-modal interaction is one possibility which

combined speech and gesture to support intuitive interaction

109