High-Fidelity Augmented Reality Interactions Hrvoje Benko - - PowerPoint PPT Presentation

▶
high fidelity augmented reality interactions
SMART_READER_LITE
LIVE PREVIEW

High-Fidelity Augmented Reality Interactions Hrvoje Benko - - PowerPoint PPT Presentation

Jan 28, 2024 192 likes •971 views

High-Fidelity Augmented Reality Interactions Hrvoje Benko Researcher, MSR Redmond New generation of interfaces Instead of interacting through indirect input devices (mice and keyboard), the user is interacting directly with the content.

slide-1
SLIDE 1
slide-2
SLIDE 2

High-Fidelity Augmented Reality Interactions

Hrvoje Benko Researcher, MSR Redmond

slide-3
SLIDE 3
slide-4
SLIDE 4

New generation of interfaces

Instead of interacting through indirect input devices (mice and keyboard), the user is interacting directly with the content. Direct un-instrumented interaction Content is the interface

slide-5
SLIDE 5

Surface computing

slide-6
SLIDE 6

Kinect

slide-7
SLIDE 7

New generation of interfaces

Direct un-instrumented interaction. Content is the interface.

slide-8
SLIDE 8

New generation of interfaces Bridge the gap between “real” and “virtual” worlds...

slide-9
SLIDE 9

… but still confined to the rectangular screen!

slide-10
SLIDE 10

Enable interactivity on any available surface and between surfaces.

An opportunity…

slide-11
SLIDE 11

MicroMotoCross

slide-12
SLIDE 12

Augmented reality

Spatial “Deviceless” High-fidelity

slide-13
SLIDE 13

Depth Sensing Cameras

slide-14
SLIDE 14

Depth sensing cameras

Color + depth per pixel: RGBZ Can compute world coordinates of every point in the image directly.

slide-15
SLIDE 15

Three basic types

  • Stereo
  • Time of flight
  • Structured light
slide-16
SLIDE 16

Correlation-based stereo cameras

Binocular disparity TZYX http://www.tyzx.com/ Point Grey Research http://www.ptgrey.com

slide-17
SLIDE 17

Correlation-based stereo

slide-18
SLIDE 18

Stereo drawbacks

  • Requires good texture to perform matching
  • Computationally intensive
  • Fine calibration required
  • Occlusion boundaries
  • Naïve algorithm very noisy
slide-19
SLIDE 19

3DV ZSense 3DV, Canesta (no-longer public) PMD Technologies http://www.PMDTec.com Mesa Technologies http://www.mesa-imaging.ch

Time of flight cameras

Pulsed infrared lasers Infrared camera + GaAs solid state shutter RGB camera

slide-20
SLIDE 20

Time of flight measurement

slide-21
SLIDE 21

Structured light depth cameras

Infrared projector RGB camera Infrared camera

http://www.primesense.com http://www.microsoft.com/kinect

slide-22
SLIDE 22

Structured light (infrared)

slide-23
SLIDE 23

Depth by binocular disparity

  • Expect a certain pattern at a given point
  • Find how far this pattern has shifted
  • Relate this shift to depth (triangulate)

Projector Camera

slide-24
SLIDE 24
  • Per-pixel depth (mm)
  • PrimeSense reference design
  • Field of View 58° H, 45° V, 70° D
  • Depth image size VGA (640x480)
  • Spatial x/y resolution (@ 2m distance from sensor) 3mm
  • Depth z resolution (@ 2m distance from sensor) 1cm
  • Operation range 0.8m - 3.5m
  • Best part – It is affordable - $150

Kinect depth camera

slide-25
SLIDE 25

Why sense with depth cameras?

Requires no instrumentation of the surface/environment. Easier understanding of physical objects in space.

slide-26
SLIDE 26

Enabling interactivity everywhere

slide-27
SLIDE 27

LightSpace

slide-28
SLIDE 28

LightSpace

slide-29
SLIDE 29

LightSpace Implementation

Projectors PrimeSense Depth Cameras

slide-30
SLIDE 30

PrimeSense depth cameras

320x240 @ 30Hz Depth from projected structured light Small overlapping areas Extended space coverage

slide-31
SLIDE 31

Unified 3D Space

slide-32
SLIDE 32

Camera & projector calibration

Projector Depth Camera

Tc Tp

Origin (0,0,0)

& Intrinsic Camera

Parameters

& Intrinsic Projector

Parameters

slide-33
SLIDE 33

Camera & projector calibration

Projector Depth Camera

slide-34
SLIDE 34

LightSpace authoring

All in real world coordinates. Irrespective of “which” depth camera. Irrespective of “which” projector.

slide-35
SLIDE 35

Supporting rich analog interactions

slide-36
SLIDE 36

Skeleton tracking (Kinect)

slide-37
SLIDE 37

Our approach

Use the full 3D mesh. Preserve the analog feel through physics-like behaviors. Reduce the 3D reasoning to 2D projections.

slide-38
SLIDE 38

Pseudo-physics behavior

slide-39
SLIDE 39

Virtual depth cameras

slide-40
SLIDE 40

Simulating virtual surfaces

slide-41
SLIDE 41

Through-body connections

slide-42
SLIDE 42

Physical connectivity

slide-43
SLIDE 43

Spatial widgets

User-aware, on-demand spatial menu

slide-44
SLIDE 44

What is missing?

LightSpace

  • “Touches” are hand

blobs

  • All objects are 2D
  • Very coarse

manipulations

Ideally

  • Multi-touch
  • 3D virtual objects
  • Full hand manipulations
slide-45
SLIDE 45

Touch on every surface

slide-46
SLIDE 46

Problem of two thresholds

Surface noise Reasonable finger thickness

slide-47
SLIDE 47

Analytically

  • Problems:
  • Slight variation in surface flatness
  • Slight uncorrected lens distortion effect in depth image
  • Noise in depth image

How to get surface distance?

slide-48
SLIDE 48

Empirically

  • Take per-pixel statistics of the empty surface
  • Can accommodate different kinds of noise
  • Can model non-flat surfaces
  • Observations:
  • Noise is not normal, nor the same at every pixel location
  • Depth resolution drops with distance

How to get surface distance?

slide-49
SLIDE 49

Build a surface histogram at every pixel.

Modeling the surface

Surface noise threshold

slide-50
SLIDE 50

Setting reasonable finger thickness

Must make some assumption about anthropometry, posture, and noise.

slide-51
SLIDE 51

Finger threshold 14mm 30mm Surface noise 3mm 6mm Camera above surface 0.75m 1.5m

How good can you get?

slide-52
SLIDE 52

KinectTouch

slide-53
SLIDE 53

But these are all static surfaces

How to allow touch on any (dynamic) surface?

  • Dynamic surface calibration
  • Tracking high-level constructs such as finger posture, 3D shape
  • Take only the ends of objects with physical extent (“fingertips”)
  • Refinement of position
slide-54
SLIDE 54

Depth camera touch sensing is almost as good as conventional touch screen technology! Works on any surface! (curved, flexible, deformable, flat…)

slide-55
SLIDE 55

Interacting with 3D objects

slide-56
SLIDE 56

Previous approaches were 2D

slide-57
SLIDE 57

Can one hold a virtual 3D object in their hand? And manipulate it using the full dexterity of your hand?

slide-58
SLIDE 58

If you know the geometry of the world, you should be able to simulate physical behaviors.

slide-59
SLIDE 59

Problems with physics and depth cameras

Dynamic meshes are difficult

  • Rarely supported in physics packages

No lateral forces!

  • Can’t place torque on an object

Penetration is handled badly

  • Can’t grasp an object with two fingers
slide-60
SLIDE 60

Particle proxy representations

slide-61
SLIDE 61

But can you see 3D in your hand?

slide-62
SLIDE 62

3D perception

Many cues:

  • Size
  • Occlusions
  • Shadows
  • Motion parallax
  • Stereo
  • Eye focus and convergence

Can correctly simulate if you know:

  • The geometry of the scene
  • User’s view point and gaze
slide-63
SLIDE 63

Can easily capture scene geometry Can easily track user’s head

Depth camera is ideal for this!

slide-64
SLIDE 64

3D Projector (Acer H5360) Shutter Glasses (Nvidia 3D Vision) Depth Camera (Kinect)

MirageBlocks

slide-65
SLIDE 65

A single user experience!

slide-66
SLIDE 66

Particle proxies

slide-67
SLIDE 67

MirageBlocks

slide-68
SLIDE 68

Next: Grabbing

Very hard problem – Working on it!

slide-69
SLIDE 69

Summary

1. Interactivity everywhere 2. Room and body as display surfaces 3. Touch and 3D interactions 4. Preserve the analog feel of interactions

slide-70
SLIDE 70

MirageBlocks demo Friday 10am – 1pm

Come to try it yourself!

slide-71
SLIDE 71

Resources to consider

slide-72
SLIDE 72

Resources

Kinect for Windows SDK

  • http://research.microsoft.com/en-

us/um/redmond/projects/kinectsdk

slide-73
SLIDE 73

NVIDIA PhysX SDK

  • http://developer.nvidia.com/physx-downloads
  • http://physxdotnet.codeplex.com/ (.NET wrappers)

Newton Physics Game Engine

  • http://newtondynamics.com/forum/newton.php

Resources

slide-74
SLIDE 74

NVIDIA 3D Vision

  • http://www.nvidia.com/object/3d-vision-main.html

DLP Link

  • http://www.dlp.com/projector/dlp-innovations/dlp-link.aspx
  • http://www.xpand.me/ (3D glasses)

Resources

slide-75
SLIDE 75

My collaborators

slide-76
SLIDE 76

http://research.microsoft.com/~benko Hrvoje Benko benko@microsoft.com

?

slide-77
SLIDE 77