Projection-Based Augmented Reality Alexander Cebulla - - PowerPoint PPT Presentation

▶
projection based augmented reality
SMART_READER_LITE
LIVE PREVIEW

Projection-Based Augmented Reality Alexander Cebulla - - PowerPoint PPT Presentation

Feb 29, 2024 129 likes •660 views

Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1 The Bigger Picture Overall topic: Interaction in Intelligent Environments. [] information and

slide-1
SLIDE 1

Projection-Based Augmented Reality

Alexander Cebulla

acebulla@student.ethz.ch 16.04.2013 1 Distributed Systems Seminar

slide-2
SLIDE 2

The Bigger Picture

Overall topic: “Interaction in Intelligent Environments“.

  • “[…] information and communication technology (ICT) disappears as

it becomes embedded into physical objects and the spaces in which we live and work.”

(Intelligent Spaces: The Application of Pervasive ICT (2006)) 16.04.2013 2 Distributed Systems Seminar

http://web.media.mit.edu/~raskar/UNC/ Office/sketch.gif

slide-3
SLIDE 3

What is Augmented Reality?

  • Augmentation of reality with virtual information
  • Information is context-dependent

16.04.2013 3 Distributed Systems Seminar

slide-4
SLIDE 4

Optical vs. Video See-Through Displays

16.04.2013 4 Distributed Systems Seminar

slide-5
SLIDE 5

Overview

16.04.2013 5 Distributed Systems Seminar

(Spatial Augmented Reality Merging Real and Virtual Worlds Oliver Bimber, Ramesh Raskar)

slide-6
SLIDE 6
  • Indoor Navigation

Earlier Talk: “Handheld Augmented Reality”

16.04.2013 6 Distributed Systems Seminar

  • Touch-free interface
  • Visualization
slide-7
SLIDE 7

Why Smart Phones?

Advantages of smart phones:

  • Relatively cheap
  • Common and available
  • Has a range of sensors already built in: GPS, accelerometer, …
  • Unlimited field of view through moving the smart phone around

16.04.2013 7 Distributed Systems Seminar

  • Hand-held video see-through displays
slide-8
SLIDE 8

Why Smart Phones are not Enough?

However:

  • Small display:
  • Limited field of view, if one does not want to move the phone

all the time

  • User might need to hold / move the device over an extended

period of time

  • Limited resolution:
  • With video mixing reality has the resolution of the camera.

16.04.2013 8 Distributed Systems Seminar

One solution:

  • Spatial optical see-through display
slide-9
SLIDE 9

Spatial Optical See-Through Display

16.04.2013 9 Distributed Systems Seminar

One possible build:

  • Combining a beam splitter with a projector or monitor.

Standard example: Virtual Showcase

Discussed in: The Virtual Showcase as a new Platform for Augmented Reality Digital Storytelling [Bimber2003]

slide-10
SLIDE 10

How is One Build?

16.04.2013 10 Distributed Systems Seminar

It is a spatial optical see- through display.

  • Two designs were discussed

(1) Convex assembly of four half-silvered mirror beam splitters (2) Projection screen a) Single CRT projector b) Four CRT monitors (3) Controllable light projector (4) Infrared emitters (5) Shutter glasses (6) Electromagnetic tracking device Virtual Showcase

(b)

slide-11
SLIDE 11

16.04.2013 11 Distributed Systems Seminar Virtual Showcase http://www.youtube.com/watch?v=qgN8T-iVRL8

slide-12
SLIDE 12

Advantages and Problems

16.04.2013 12 Distributed Systems Seminar

Advantages:

  • Provides a high resolution
  • Even higher with the monitor-based approach.
  • Increased control over environmental factors (lighting, sound, …)

inside and outside the Virtual Showcase.

  • Example: Creation of realistic occlusion effects through pixel-

wise illumination of physical content.

  • Easier eye accommodation (usually a problem with optical see-

through displays, which are distant from the augmented object)

  • Furthermore: Larger screen size

Virtual Showcase

Problems:

  • Limitations in multi-user support
  • currently only four users can be supported simultaneously
  • Useless for mobile applications

“Retina” displays + HD camera Video mixing Price:

slide-13
SLIDE 13

Overview

16.04.2013 13 Distributed Systems Seminar

(Spatial Augmented Reality Merging Real and Virtual Worlds Oliver Bimber, Ramesh Raskar)

slide-14
SLIDE 14

What are the Main Advantages of Projected AR?

16.04.2013 14 Distributed Systems Seminar

A projector

  • allows planar displays of various shapes.
  • allows non-planar, curved displays.
  • Can directly project onto physical objects
  • vercomes the restriction that display size dictates device size.
  • Unlimited field of view
  • Can be transported easily compared to large screens.
slide-15
SLIDE 15

How to Make Use of These Advantages?

16.04.2013 15 Distributed Systems Seminar

Identified problems:

  • Self-contained:
  • Little to no dependence on infrastructure.
  • Geometric-aware:
  • Understanding of the geometry of display surface.
  • Orientation of projector
  • Self-configuring:
  • Ad-hoc integration into a cluster of projectors.

Possible solution: iLamps

Discussed in: iLamps: Geometrically Aware and Self-Configuring Projectors

  • Can be transported easily compared to large screens.
  • Can directly project onto physical objects
slide-16
SLIDE 16

Creation of an Self-Contained Projector

16.04.2013 16 Distributed Systems Seminar

Add:

iLamps

  • Sensors
  • Ability to communicate with objects and other devices
  • Computing and an interface
slide-17
SLIDE 17

Geometric Awareness Through Structured Light

16.04.2013 17 Distributed Systems Seminar

Method:

1. A known light pattern (e.g. parallel stripes, random dots) is projected

  • nto the scene.

2. From the deformation of the pattern, depth and surface information can be computed.

Ajmal Saeed Mian, Mohammed Bennamoun and Robyn Owens: 3D model-based free-form object recognition – a review

slide-18
SLIDE 18

How to Adapt to Unideal Display Shapes?

16.04.2013 18 Distributed Systems Seminar

Problem:

  • Use of surfaces like room corners or columns.
  • Regardless of the viewpoint, the distortion of the projected image

should be minimized.

iLamps

A possible solution:

  • Measure distortion with conformality (angle preserving)

1. Use the camera to capture images of projected structured light. 2. Generate 3D mesh. 3. Compute a conformal map, which is used as texture map.

slide-19
SLIDE 19

How to Adapt to Unideal Display Shapes?

16.04.2013 19 Distributed Systems Seminar iLamps http://www.youtube.com/watch?v=7IdHTt6pCjY

slide-20
SLIDE 20

Problems

16.04.2013 20 Distributed Systems Seminar

Possible Solution:

  • So far: Only one projector

Cluster of projectors (iLamps)

  • Field of view is still limited
  • Low light intensity
slide-21
SLIDE 21

Cluster of Projectors

16.04.2013 21 Distributed Systems Seminar

Does the new projector belong to group?

  • ‘Ready to join’ msg. + light pattern
  • If seen by any other camera, then quick calibration step.

iLamps

Quick calibration

  • Sequentially: Projection of structured pattern (checkerboard)
  • By each unit (i.e. in parallel):
  • Computation of the largest inscribed rectangle.
  • Checking how own projection fits into this rectangle.
  • Projection of corresponding image part.
slide-22
SLIDE 22

Cluster of Projectors

16.04.2013 22 Distributed Systems Seminar iLamps http://www.youtube.com/watch?v=NfB5qwD9z9s

slide-23
SLIDE 23

Problems

16.04.2013 23 Distributed Systems Seminar

Possible Solutions:

  • So far: Only one projector
  • 1. Cluster of projectors (iLamps)
  • Field of view is still limited
  • Low light intensity
  • Remember: Hand-held displays can be moved around …
  • 2. Steerable displays (Beamatron)
slide-24
SLIDE 24

Steerable Displays

16.04.2013 24 Distributed Systems Seminar Discussed in: Steerable augmented reality with the beamatron

Problems:

  • Geometric-aware:
  • Project images ‘correctly’ on arbitrary surfaces
  • Stabilize projected graphics during movement.
  • Track the user position

The Beamatron:

slide-25
SLIDE 25

Stabilize Projected Graphics During Movement

16.04.2013 25 Distributed Systems Seminar

Solution:

  • Build a circuit board which directly connects to the built-in pan and tilt

sensors of the platform.

Problem:

  • The light platform does not provide any feedback on the pan and tilt

configuration.

  • Real-time knowledge of the platform pose is critical for projecting

stabilized graphics.

The Beamatron

slide-26
SLIDE 26

Understanding the Geometry of the Display Surface

16.04.2013 26 Distributed Systems Seminar

  • Use the mounted Kinect sensor with Kinect Fusion to obtain smoothed

depth images:

The Beamatron

slide-27
SLIDE 27

Project images ‘correctly’ on arbitrary surfaces

16.04.2013 27 Distributed Systems Seminar

Already have seen

  • ‘Wall-paper’ an image onto a real object.

Now:

  • Create the illusion of a real 3D Object.

Requires two rendering passes:

1) Render the real objects along with virtual objects from the point of view of the user. 2) Use the result as texture map for the rendering of the real geometry.

The Beamatron

slide-28
SLIDE 28

Track the User Position

16.04.2013 28 Distributed Systems Seminar

Solution:

  • Use the array microphone of the Kinect sensor to localize the user as

they speak.

  • Also useful for recognizing commands.
  • Three Kinect sensors are mounted in the corners of the room.

Two horizontally and one vertical.

  • One sensor covers a

range of around 100°

Problem:

  • Hard to monitor events outside of the field of view of the Kinect sensor.

The Beamatron

slide-29
SLIDE 29

Application: Beamabuggy

16.04.2013 29 Distributed Systems Seminar The Beamatron http://www.youtube.com/watch?v=Z4bdrG8S1FM

slide-30
SLIDE 30

Overview

16.04.2013 30 Distributed Systems Seminar

(Spatial Augmented Reality Merging Real and Virtual Worlds Oliver Bimber, Ramesh Raskar)

slide-31
SLIDE 31

Going Mobile

16.04.2013 31 Distributed Systems Seminar

Until now:

  • Orientation of the projector
  • Shape of the real-object
  • Cluster: Position relative to other projectors
  • Geometric-awareness

Now:

  • Position of the projector relative to the scene
slide-32
SLIDE 32

Going Mobile

16.04.2013 32 Distributed Systems Seminar

Idea:

  • Spatial awareness: System infers location and orientation of the

device in 3D space.

  • Geometry awareness: System constructs the 3D structure of the

world (User, Furniture, Walls, …) around it.

Two approaches:

1) Infrastructure-based: Uses sensors positioned in the environment. 2) Infrastructure-less: All sensors are attached to the projector.

Discussed in: Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces

slide-33
SLIDE 33

Going Mobile

16.04.2013 33 Distributed Systems Seminar iLamps

Use of markers allows:

  • computation of camera and hence projector pose relative to the

markers.

  • identifying individual objects.
slide-34
SLIDE 34

Going Mobile

16.04.2013 34 Distributed Systems Seminar iLamps http://www.youtube.com/watch?v=yJZLIHBr1G0

slide-35
SLIDE 35

Infrastructure-Based: RoomProjector

16.04.2013 35 Distributed Systems Seminar

The Projector:

  • Inertial Measurement Unit (IMU)
  • Orientation of the device
  • IR Camera

In the room:

  • Four Kinect cameras mounted at the ceiling at the mid-point of each

wall.

Discussed in: Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces

slide-36
SLIDE 36

Register the Environment

16.04.2013 36 Distributed Systems Seminar

When the room is empty:

  • For each Kinect sensor the average of a number of depth map samples

is computed.

  • From these averages a single fused point cloud is generated.
  • The point cloud is used to produce a background mesh.
  • This mesh is used for background subtraction.

RoomProjector

slide-37
SLIDE 37

Track the Projector Position

16.04.2013 37 Distributed Systems Seminar

Absolute position in the room:

  • Cover the projector with retro-reflective tape:
  • Light pattern from the Kinect will be reflected and appears

much brighter in the 2D Kinect IR image.

  • The position of the projector can then be triangulated.

RoomProjector

Orientation of the projector:

  • Provided by the Inertial Measurement Unit
slide-38
SLIDE 38

Project Images ‘Correctly’ on Arbitrary Surfaces

16.04.2013 38 Distributed Systems Seminar

Mesh of the room + Absolute position in the room + Orientation

  • Automatic correction of projected content.

RoomProjector

Furthermore:

  • Flashlight-like metaphor: Can associate 2D image with any

surface and use the projector to “reveal” it.

slide-39
SLIDE 39

Interaction

16.04.2013 39 Distributed Systems Seminar RoomProjector

  • Use of a hot (IR) mirror: IR camera and projector are coaxial.
  • Can segment hand of user.
  • Interactions happen in the coordinate space of the camera.

http://www.youtube.com/watch?v=frGEzlrhve0

slide-40
SLIDE 40

Interaction

16.04.2013 40 Distributed Systems Seminar RoomProjector

Combine data from IR camera and Kinect sensors:

http://www.youtube.com/watch?v=frGEzlrhve0

slide-41
SLIDE 41

Interaction: Objects in Mid-Air

16.04.2013 41 Distributed Systems Seminar RoomProjector

  • Virtual image projected onto the surface
  • How to render virtual objects, which are in mid-air?
  • Project shadows
  • Use a mobile plane as viewport into the virtual world
slide-42
SLIDE 42

Interaction: Objects in Mid-Air

16.04.2013 42 Distributed Systems Seminar RoomProjector http://www.youtube.com/watch?v=frGEzlrhve0

slide-43
SLIDE 43

Advantages and Problems

16.04.2013 43 Distributed Systems Seminar RoomProjector

Advantages:

  • Small sensing window of the projector is overcome.

Problem: Coarseness

  • Only prominent objects can be recovered from the scene.
  • Hybrid tracking of projector can be noisy and error prone:
  • camera occlusion
  • ferrous objects interfering with the IMU
slide-44
SLIDE 44

Infrastructure-Less: SLAMProjector

16.04.2013 44 Distributed Systems Seminar

  • KinectFusion uses its depth data to:
  • Building a reconstruction of the environment in real-time
  • Geometry awareness
  • Recover the pose of the projector
  • Spatial awareness

SLAM: Simultaneous Localization and Mapping

  • Kinect sensor is directly mounted on the projector.
slide-45
SLIDE 45

Interaction

16.04.2013 45 Distributed Systems Seminar SLAMProjector

  • Interaction similar to those of the RoomProjector are possible.
  • Furthermore, can move Kinect to object of interest:
  • Higher resolution:
  • Possible to interact with less prominent objects
  • Can segment and copy them
  • No camera occlusion.
slide-46
SLIDE 46

Interaction

16.04.2013 46 Distributed Systems Seminar SLAMProjector http://www.youtube.com/watch?v=frGEzlrhve0

slide-47
SLIDE 47

Advantages and Problems

16.04.2013 47 Distributed Systems Seminar RoomProjector

Advantages:

  • High 3D sensing fidelity
  • Independent from infrastructure

Problem:

  • Shadow interaction: User occludes a large part of the Kinect depth
  • image. Can degrade the tracking quality of the projector.
  • Location is tracked from the built model, accumulated errors can

lead to drift.

slide-48
SLIDE 48

Summary

16.04.2013 48 Distributed Systems Seminar

Spatial optical see-through display:

  • Virtual Showcase

Main advantages:

  • Can directly project onto physical objects of various shapes
  • Overcomes the restriction that display size dictates device size.
  • Unlimited field of view
  • Can be transported easily compared to large screens.
slide-49
SLIDE 49

Summary

16.04.2013 49 Distributed Systems Seminar

How to Make Use of These Advantages?

  • iLamps:
  • Self-contained, Geometric-aware and Self-organizing
  • Disadvantages:
  • Limited field of view
  • Light intensity

Solutions:

  • Cluster of projectors
  • Steerable displays
  • Beamatron
slide-50
SLIDE 50

Summary

16.04.2013 50 Distributed Systems Seminar

Going mobile:

  • Geometry AND spatial aware
  • Infrastructure-based:
  • iLamp: Markers
  • Only discrete number of surfaces
  • RoomProjector: 4 Kinect sensors
  • Advantage: Extended sensing
  • Disadvantage: Coarse
  • Infrastructure-less:
  • SLAMProjector: Kinect sensor mounted on projector
  • Advantage: High 3D sensing fidelity
  • Disadvantage: Drift of camera position
  • Problem when displaying virtual objects in mid-air.
slide-51
SLIDE 51

Future

16.04.2013 51 Distributed Systems Seminar

Combination of smart phones with projectors:

slide-52
SLIDE 52

Future

16.04.2013 52 Distributed Systems Seminar

Augmenting a complete room:

http://www.youtube.com/watch?v=N_cKsOe7hLI