VU Augmented Reality on Mobile Devices VU Augmented Reality on - - PowerPoint PPT Presentation

▶
vu augmented reality on mobile devices vu augmented
SMART_READER_LITE
LIVE PREVIEW

VU Augmented Reality on Mobile Devices VU Augmented Reality on - - PowerPoint PPT Presentation

Mar 03, 2023 163 likes •1.01k views

VU Augmented Reality on Mobile Devices VU Augmented Reality on Mobile Devices Introduction Introduction What is AR What is AR Interaction Techniques Navigation Collaboration Navigation, Collaboration Visualization

slide-1
SLIDE 1

VU Augmented Reality on Mobile Devices VU Augmented Reality on Mobile Devices

  • Introduction

What is AR

  • Introduction – What is AR
  • Interaction Techniques
  • Navigation Collaboration

Navigation, Collaboration

  • Visualization Techniques
  • Visual Coherence

Visual Coherence

  • Tracking
  • …

Includes material from Denis Kalkofen, TU Graz,

1

, ,

Georg Klein Oxford, Jan Fischer

slide-2
SLIDE 2

Augmented Graphics Augmented Graphics

  • Information
  • Visualization
  • Understanding
  • Virtual Objects
  • Graphics
  • Realism

2

Understanding Realism

slide-3
SLIDE 3

Topics Topics

  • Occlusion
  • Occlusion
  • Lighting

Lighting

  • Camera effects

Camera effects

  • Visual effects

3

3

slide-4
SLIDE 4

Scene models

  • Computer graphics completely models real scene

Scene models

  • Computer graphics completely models real scene
  • Geometry accessible

Geometry accessible for calculations

  • Occlusions

Li hti

  • Lighting
  • Shadows
  • Camera parameters

4

4

slide-5
SLIDE 5

Augmented Reality Augmented Reality

  • Less information

available

  • Final image through

iti compositing

  • Digital in video see-through
  • Physical in optical see-

through, projection

  • Approximate real world
  • Approximate real world

to simulate interaction

  • geometrical, appearance

5

5

  • post processing
slide-6
SLIDE 6

Occlusion Handling Occlusion Handling

  • Virtual in front of real
  • Draw augmentation on top of

video background video background

  • Virtual behind real
  • Need strategy to distinguish

i ibl f l d d

QuickTime™ and a decompressor are needed to see this picture.

visible from occluded augmentations

6

slide-7
SLIDE 7

Phantom Rendering Phantom Rendering

  • Render registered virtual representations

g p (Phantoms) of real objects

  • Occlusions handled by

hi h d graphics hardware

1.Draw Video 2.Disable writing to color buffer (glColorMask or glBlendFunc(0,1)) 3 Render phantoms of 3.Render phantoms of real scene ➔ sets depth buffer 4.Enable writing to color buffer

7

7

5.Draw virtual objects

slide-8
SLIDE 8

Phantom Rendering Phantom Rendering

8

slide-9
SLIDE 9

Phantom Rendering Phantom Rendering

  • Polygonal data [Breen96 Fiala06]
  • Polygonal data [Breen96,Fiala06]

9

slide-10
SLIDE 10

Phantom Rendering Phantom Rendering

  • Volume data
  • Volume data
  • Phantom = Isosurface using ’FirstHit Raycasting’

10

[Fischer04]

slide-11
SLIDE 11

Problems of Phantom Rendering Problems of Phantom Rendering

  • Requires accurate
  • Requires accurate
  • Model
  • Tracking data

R i t ti

  • Registration

11

slide-12
SLIDE 12

Occlusion Refinement Occlusion Refinement

  • To overcome poor
  • To overcome poor
  • Model
  • Tracking

R i t ti

  • Registration

12

[DiVerdi06]

slide-13
SLIDE 13

Occlusion Refinement Occlusion Refinement

  • Image based post processing
  • Image based post-processing

[DiV di06] [Kl i 05]

13

[DiVerdi06] [Klein05]

slide-14
SLIDE 14

Occlusion Refinement Occlusion Refinement

14

slide-15
SLIDE 15

Probabilistic Occlusion Handling Probabilistic Occlusion Handling

  • Soft transition between occluding and hidden objects
  • Soft transition between occluding and hidden objects
  • Compensate for tracking and registration error, by

reducing the occluder’s transparency depending on g p y p g the probability of occlusion

15

[Fuhrmann99]

slide-16
SLIDE 16

Model Free Occlusion Handling Model Free Occlusion Handling

  • Instead of tracked & registered

g phantom model

  • Construct depth map from video
  • from Computer Vision
  • Stereo, Shading, Structured Light etc.
  • Consider Performance

Co s de e o a ce

16

[Fischer07] [PointGrey Bumblebee]

slide-17
SLIDE 17

Modell Free Occlusion Handling Modell Free Occlusion Handling

  • Initialize rendering of augmentations with depth map
  • Initialize rendering of augmentations with depth map

from video

17

[Fischer07]

slide-18
SLIDE 18

Model Free Occlusion Handling Model Free Occlusion Handling

  • Using a priori

g p knowledge about scene layout

  • Foreground and

Foreground

  • Foreground and

background objects are known

  • Online separation by
  • Online separation by

foreground illumination

[Grundhoefer07]

18

slide-19
SLIDE 19

Model Free Occlusion Handling Model Free Occlusion Handling

  • Online separation by foreground illumination
  • Online separation by foreground illumination

19

[Grundhoefer07]

slide-20
SLIDE 20

Model Free Occlusion Handling Model Free Occlusion Handling

  • Using a priori knowledge

g p g about scene layout

  • Hands are always in front and

visible

Hand Occlusion Hand Occlusion

visible

  • Hand segmentation by

using hand color

Hand Occlusion Hand Occlusion [Sandor07] [Sandor07]

20

slide-21
SLIDE 21

Semi-Automatic Model Free Occlusion Handling Handling

  • Manual a priori foreground segmentation
  • Manual a priori foreground segmentation
  • Boundary tracking

Boundary

21

[Lepetit00] Boundary

slide-22
SLIDE 22

Lighting Lighting

  • Shadows
  • Shadows
  • Global illumination

Global illumination

  • Interaction between real and virtual objects

Interaction between real and virtual objects

22

slide-23
SLIDE 23

Shadows Shadows

  • Important for
  • Important for
  • Depth perception
  • Scene interpretation
  • Relations
  • real ➔ real

ea ea

  • real ➔ virtual
  • virtual ➔ real
  • virtual ➔ virtual

virtual ➔ virtual

[Haller03]

23

[Haller03]

slide-24
SLIDE 24

Scene model Scene model

  • Virtual objects

Virtual objects

  • Impostors for

pos o s o real objects

  • Light sources
  • real must be reflected

in shading

  • real must be used to

shadow casting

24

slide-25
SLIDE 25

Drop shadow on plane Drop shadow on plane

  • Simple scene geometry
  • Simple scene geometry
  • Only virtual shadows

25

slide-26
SLIDE 26

Shadow volume rendering Shadow volume rendering

  • Multi pass shadow volume
  • Multi-pass shadow volume

1.Draw real scene from video image 2.Draw virtual ➔ real shadows 3 D i t l bj t 3.Draw virtual objects 4.Draw real, virtual ➔ virtual objects

26

slide-27
SLIDE 27

Illumination Illumination

  • Real scenes have complex illumination
  • Real scenes have complex illumination
  • More than 1 light source
  • Complex light sources - area, inter-reflection, color

Gl b l ill i ti th h i di t li ht

  • Global illumination through indirect light
  • Measure illumination
  • Measure illumination
  • Light probe

27

slide-28
SLIDE 28

Light probe for environment maps Light probe for environment maps

  • Light probe
  • Shiny crome sphere
  • Create HDR
  • Create HDR

environment map

  • Captures most directions

28

[Devebec98]

slide-29
SLIDE 29

Image-based Lighting and shadows Image-based Lighting and shadows

  • Illuminate

surfaces with environment map map

[Devebec98]

  • Approximate

with light field

[ ]

g and blend multiple shadows

29

shadows

[Knecht10]

slide-30
SLIDE 30

Image-based Lighting and shadows Image-based Lighting and shadows

30

slide-31
SLIDE 31

Light acquisition without a sphere Light acquisition without a sphere

  • Calibrate multi-camera system
  • planar target with feature point

[Pilet06]

  • planar target with feature point

recognition

O li d ff li ti ti

  • On-line and off-line estimation
  • f a light map
  • no light occlusions

31

  • depends only on normal direction
slide-32
SLIDE 32

Now all together Now all together

  • Interaction between

close real and virtual

  • bjects
  • light bleeding
  • light bleeding
  • shadowing
  • refraction
  • Render dynamic

environment maps per p p virtual object

(a) Environment map (b) Light sources (c) Virtual objects

32

(d) Impostors [Pessoa10]

slide-33
SLIDE 33

Now all together Now all together

33

slide-34
SLIDE 34

Camera Effects Camera Effects

  • Small & cheap cameras & lenses
  • Small & cheap cameras & lenses
  • Many problems
  • Distortion
  • Vignetting
  • Chromatic aberration
  • Softness in corners
  • Bayer pattern artifacts
  • Color compression
  • Arbitrary image processing

Arbitrary image processing

  • Motion blur

34

f=2.1 mm, f/2.0, around $10

slide-35
SLIDE 35

35

A terrible image

slide-36
SLIDE 36

36

Graphics don’t match

slide-37
SLIDE 37

Lens distortion Lens distortion

  • Wide angle lenses

g

  • good for tracking
  • not well modeled with a pinhole

camera

  • Texture-based Approach
  • Watson & Hodges ’95
  • Render undistorted initially
  • Use texture mapping to distort

pp g rendered image

  • Geometry-based Approach

37

Geometry based Approach

  • Vertex shader to tweak geometry
slide-38
SLIDE 38

Render un-distorted visuals Render un-distorted visuals

  • Render into off screen

buffer

[Klein05]

  • Prepare a grid for

texture mapping

38

texture mapping

slide-39
SLIDE 39

Compositing with video background Compositing with video background

  • Draw video background

[Klein05]

  • Overlay distorted

texture

39

texture

slide-40
SLIDE 40

40

Final Image

slide-41
SLIDE 41

Geometry-based Approach Geometry-based Approach

  • Vertex shader to move vertices in projected image
  • Requires geometry subsampling

41

q g y p g

  • Artifacts at the image edge
slide-42
SLIDE 42

More effects More effects

  • Lens
  • Lens
  • Vignetting
  • Chromatic aberration

S ft i

Sony 1/4” CCD 640x480@30Hz

  • Softness in corners
  • Image acquisition

640x480@30Hz Bayer mask

g q

  • Antialiasing
  • Sharping
  • Quantization

Quantization

  • Bayer CFA artifacts
  • Color compression

42

slide-43
SLIDE 43

Emulating camera effects Emulating camera effects

  • Some effects require changes to 3D rendering
  • Some effects require changes to 3D rendering
  • E.g: Depth defocus, proper motion blur
  • Expensive !

p

  • Focus on post-processing:
  • Take single RGBA image rendered in OpenGL
  • Modify this image to emulate camera effects
  • Limitations

Limitations

  • No full motion blur

43

[Klein08]

slide-44
SLIDE 44

640x480 image from fire 640x480 image from fire-

  • i

i A tracking system provides pose and velocity estimates A tracking system provides pose and velocity estimates 640x480 image from fire 640x480 image from fire-

  • i

i A tracking system provides pose and velocity estimates A tracking system provides pose and velocity estimates A tracking system provides pose and velocity estimates A tracking system provides pose and velocity estimates A tracking system provides pose and velocity estimates A tracking system provides pose and velocity estimates

44

  • 0. Input image
slide-45
SLIDE 45

3000x2250 pixels 3000x2250 pixels Perfect pinhole projection model Perfect pinhole projection model 3000x2250 pixels 3000x2250 pixels Perfect pinhole projection model Perfect pinhole projection model Perfect pinhole projection model Perfect pinhole projection model 8-

  • bit RGBA with

bit RGBA with pre pre‐multiplied alpha multiplied alpha Perfect pinhole projection model Perfect pinhole projection model 8-

  • bit RGBA with

bit RGBA with pre pre‐multiplied alpha multiplied alpha

45

  • 1. Rendered Image
slide-46
SLIDE 46

Distort image with texture Distort image with texture-

  • mapped grid

mapped grid Output 2048x1536 Output 2048x1536 Distort image with texture Distort image with texture-

  • mapped grid

mapped grid Output 2048x1536 Output 2048x1536

46

  • 2. Apply lens distortion
slide-47
SLIDE 47

Subsample by 2:1 with 4x4 box filter Subsample by 2:1 with 4x4 box filter Cross Cross-

  • mix 10% of R,G,B values

mix 10% of R,G,B values Subsample by 2:1 with 4x4 box filter Subsample by 2:1 with 4x4 box filter Cross Cross-

  • mix 10% of R,G,B values

mix 10% of R,G,B values , , , , Output image 1024x768 pixels Output image 1024x768 pixels , , , , Output image 1024x768 pixels Output image 1024x768 pixels

47

  • 3. Half-sample and colour mix
slide-48
SLIDE 48

Two Two-

  • pass separable convolution

pass separable convolution Blur magnitude varies across image Blur magnitude varies across image Two Two-

  • pass separable convolution

pass separable convolution Blur magnitude varies across image Blur magnitude varies across image Blur magnitude varies across image Blur magnitude varies across image (Corner softness & motion blur) (Corner softness & motion blur) Blur magnitude varies across image Blur magnitude varies across image (Corner softness & motion blur) (Corner softness & motion blur)

48

  • 4. 7-tap Gaussian Blur
slide-49
SLIDE 49

Emulate camera rotation only Emulate camera rotation only Emulate camera rotation only Emulate camera rotation only

49

  • 5. 16-tap motion blur
slide-50
SLIDE 50

From 1024x768 image, sample three From 1024x768 image, sample three images: images: From 1024x768 image, sample three From 1024x768 image, sample three images: images: Red+Alpha Red+Alpha 320x240 320x240 Green+Alpha Green+Alpha 559x560 559x560 Blue+Alpha Blue+Alpha 320x240 320x240 Red+Alpha Red+Alpha 320x240 320x240 Green+Alpha Green+Alpha 559x560 559x560 Blue+Alpha Blue+Alpha 320x240 320x240 Blue Alpha Blue Alpha 320x240 320x240 This emulates the Bayer mask This emulates the Bayer mask Sampling is slightly distorted to emulate Sampling is slightly distorted to emulate Ch ti Ab ti Ch ti Ab ti Blue Alpha Blue Alpha 320x240 320x240 This emulates the Bayer mask This emulates the Bayer mask Sampling is slightly distorted to emulate Sampling is slightly distorted to emulate Ch ti Ab ti Ch ti Ab ti Chromatic Aberration Chromatic Aberration Also add image noise Also add image noise Chromatic Aberration Chromatic Aberration Also add image noise Also add image noise

50

  • 6. Bayer sampling
slide-51
SLIDE 51

Red+Alpha Red+Alpha 320x240 320x240 Red+Alpha Red+Alpha 320x240 320x240

51

Red+Alpha Red+Alpha 320x240 320x240 Red+Alpha Red+Alpha 320x240 320x240

  • 6. Bayer sampling
slide-52
SLIDE 52

Green+Alpha Green+Alpha 559x560 559x560 Green+Alpha Green+Alpha 559x560 559x560

52

Green+Alpha Green+Alpha 559x560 559x560 Green+Alpha Green+Alpha 559x560 559x560

  • 6. Bayer sampling
slide-53
SLIDE 53

Blue+Alpha Blue+Alpha 320x240 320x240 Blue+Alpha Blue+Alpha 320x240 320x240

53

Blue+Alpha Blue+Alpha 320x240 320x240 Blue+Alpha Blue+Alpha 320x240 320x240

  • 6. Bayer sampling
slide-54
SLIDE 54

Bayer Samples Bayer Samples

  • Raw bayer image available?

Bl d th d B d ! Blend now, then de-Bayer - done!

54

slide-55
SLIDE 55

640x480 composite 640x480 composite 640x480 composite 640x480 composite 640x480 composite 640x480 composite Video in is YUV Video in is YUV-

  • 411

411 Render is YUV Render is YUV-

  • 444

444 640x480 composite 640x480 composite Video in is YUV Video in is YUV-

  • 411

411 Render is YUV Render is YUV-

  • 444

444

55

YUV Composite

slide-56
SLIDE 56

56

Chroma Squash

slide-57
SLIDE 57

57

Chroma Squash

slide-58
SLIDE 58

58

Finished Composite

slide-59
SLIDE 59

59

Old Method

slide-60
SLIDE 60

60

slide-61
SLIDE 61

61

slide-62
SLIDE 62

62

slide-63
SLIDE 63

63

slide-64
SLIDE 64

64

slide-65
SLIDE 65

65

slide-66
SLIDE 66

66

slide-67
SLIDE 67

Visual Effects Visual Effects

  • Realism difficult ➔ Change reality instead
  • Realism difficult ➔ Change reality instead
  • Diminished reality

Diminished reality

  • Hide real objects
  • Instrumentations

A ti ti ff t

[Mann01]

  • Artistic effects
  • Cartoon style
  • Abstraction

67

[Fischer06]

slide-68
SLIDE 68

Diminished reality Diminished reality

  • Remove fiducial

markers

  • Texture

synthesis from area around the area around the marker

  • Repeats band
  • f video

backgro nd

68

background

[Siltanen06]

slide-69
SLIDE 69

Diminished reality Diminished reality

  • Remove fiducial

markers

  • Texture

synthesis from area around the area around the marker

  • Repeats band
  • f video

backgro nd

69

background

[Siltanen06]

slide-70
SLIDE 70

Diminished Reality Diminished Reality

  • Block/remove objects in the real environment
  • Block/remove objects in the real environment
  • Haptic device removed with image-based rendering

70

[Cosco09]

slide-71
SLIDE 71

Diminished Reality Diminished Reality

71

slide-72
SLIDE 72

Diminished reality Diminished reality

72

slide-73
SLIDE 73

Stylized Augmented Reality Stylized Augmented Reality

  • Non photorealistic display of both the camera image
  • Non-photorealistic display of both the camera image

and virtual objects.

  • ➔ Adapt levels of realism

p

73

Conventional AR Stylized Augmented Reality

slide-74
SLIDE 74

Stylized AR Pipeline Stylized AR Pipeline

  • Cartoon-like stylization of augmented reality images

Requirements:

  • Interactive frame rate
  • Similar-looking output images
  • Customizable

74

Customizable

slide-75
SLIDE 75

Color Segmentation Color Segmentation

  • Aim: Smooth similarly colored regions but preserve
  • Aim: Smooth similarly colored regions, but preserve

high contrast edges

  • Bilateral filtering [Tomasi and Manduchi 1998]

g [ ]

75

  • Takes spatial distance and signal difference into account
slide-76
SLIDE 76

Non-Photorealistic Renderer Non-Photorealistic Renderer

  • Cartoon like rendering of virtual objects
  • Cartoon-like rendering of virtual objects
  • Similarity to processed camera image
  • Uniformly colored regions enclosed by thick

Uniformly colored regions enclosed by thick silhouettes

  • Two-pass rendering algorithm [Lander 2000]

76

Conventional Stylized

slide-77
SLIDE 77

Non-Photorealistic Renderer Non-Photorealistic Renderer

77

slide-78
SLIDE 78

Reality Tooning Reality Tooning

  • Redesigned algorithm: Pure postprocessing
  • Redesigned algorithm: Pure postprocessing

approach

  • Implementation on the GPU (OpenGL Shading

p ( p g Language)

  • Typically more

th 25 f than 25 fps

  • Better visual

quality quality

78

slide-79
SLIDE 79

Selective Stylization Selective Stylization

  • Problem: Entire image is stylized
  • Problem: Entire image is stylized
  • Dissociation of user from observed scene?
  • Why have AR at all ?
  • Idea: Select regions where stylization is useful
  • Depends on application case

79

slide-80
SLIDE 80

Tangible Interaction Setup Tangible Interaction Setup

80

slide-81
SLIDE 81

Graphical Augmentations Graphical Augmentations

81

slide-82
SLIDE 82

Stylization Domain - Video Stylization Domain - Video

82

slide-83
SLIDE 83

Selective Stylization - Video Selective Stylization - Video

83

slide-84
SLIDE 84

Open problems Open problems

  • Online capture
  • Scene geometry
  • Dynamic Lighting
  • Dynamic objects
  • Mobile ?

O ti l th h ?

  • Optical see-through ?
  • Occlusion HMD
  • Maybe a blending display?
  • I didn’t mention

84

  • Projection displays
  • Spatial augmented reality