Production framework for full panoramic scenes with photo-realistic - - PowerPoint PPT Presentation

production framework for full panoramic scenes with photo
SMART_READER_LITE
LIVE PREVIEW

Production framework for full panoramic scenes with photo-realistic - - PowerPoint PPT Presentation

INSTITUTE OF PURE AND APPLIED MATHEMATICS Production framework for full panoramic scenes with photo-realistic augmented reality Dalai Felinto , Aldo Zang and Luiz Velho Fisheries Centre, UBC - Vancouver, Canada Visgraf


slide-1
SLIDE 1

INSTITUTE OF PURE AND APPLIED MATHEMATICS

Production framework for full panoramic scenes with photo-realistic augmented reality

Dalai Felinto⋆, Aldo Zang† and Luiz Velho†

⋆ Fisheries Centre, UBC - Vancouver, Canada † Visgraf Laboratory, IMPA - Rio de Janeiro, Brazil

October 2012

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 1 / 34

slide-2
SLIDE 2

Outline

1

Part 1 Introduction Technique pipeline

Environment capture

2

Part 2 Rendering process

Scene primitives Shadows and reflections accounts

3

Part 3 Results Conclusions

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 2 / 34

slide-3
SLIDE 3

Introduction From a full panoramic HDR image we reconstruct the real scene, insert and illuminate synthetic elements in a photorealistic way using accurate shadows and reflections.

Real Panorama → Processing → Augmented Panorama

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 3 / 34

slide-4
SLIDE 4

Technique pipeline

1

Environment capture and calibration

2

Scene modeling

3

Scene depth image computation

4

Illumination setup

5

Synthetic elements

6

Integration and rendering

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 4 / 34

slide-5
SLIDE 5

Environment Capture Space partition for Nikon 10.5mm: 5 photos around z- axis, 1 zenith photo and 1 nadir photo. Capture equipment: Manfrotto Pano Head, Nikon DX2S camera and Nikon 10.5mm fisheye lens.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 5 / 34

slide-6
SLIDE 6

Panorama Stitching Stitching of multiple images Stitched and blended image

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 6 / 34

slide-7
SLIDE 7

Panorama Calibration

Advantages of a calibration system:

1

Allow to move important sampling regions off the poles.

2

Less concern on tripod alignment for picture capturing.

3

It works with panoramas obtained from the internet.

4

Optimal aligned axis for the world reconstruction.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 7 / 34

slide-8
SLIDE 8

Panorama Calibration

Calibration system

The four points on floor determines the orientations axis for the

  • environment. The blue, red and green lines determines the xy

(horizon), xz and yz slice planes.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 8 / 34

slide-9
SLIDE 9

Panorama Calibration

Calibration account

Orientation: x = ( p0 × p1) × ( p3 × p2) y = ( p1 × p2) × ( p0 × p3) z = x × y y = z × x We need, however, to gather more data - as the orientation alone is not sufficient!

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 9 / 34

slide-10
SLIDE 10

Panorama Calibration

Calibration account

Orientation: x = ( p0 × p1) × ( p3 × p2) y = ( p1 × p2) × ( p0 × p3) z = x × y y = z × x We need, however, to gather more data - as the orientation alone is not sufficient! Any arbitrary positive non-zero value for the camera height will produce a different (scaled) reconstruction of the original geometry in the 3d world. We have a dual system with the rectangle dimensions and the camera

  • height. The user can decide which one is the most accurate data

available.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 9 / 34

slide-11
SLIDE 11

Illumination Setup: HDR environment map EV = 0 EV = -2 EV = -4 EV = -6 Full panoramic image in HDR format.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 10 / 34

slide-12
SLIDE 12

Environment modeling

1

Used to compute scene depth, for visibility and shadows

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 11 / 34

slide-13
SLIDE 13

Environment modeling

1

Used to compute scene depth, for visibility and shadows

2

Provides the light position in world space for the reflection rays

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 11 / 34

slide-14
SLIDE 14

Environment modeling

1

Used to compute scene depth, for visibility and shadows

2

Provides the light position in world space for the reflection rays

3

The modeled environment serves as support surfaces for shadows and reflections rendering of synthetic elements

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 11 / 34

slide-15
SLIDE 15

Environment modeling Modeled base geometry (wireframe)

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 12 / 34

slide-16
SLIDE 16

Environment modeling Modeled base geometry (polygons)

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 12 / 34

slide-17
SLIDE 17

Environment modeling: Depth image of the scene Depth image reconstruction from base geometry

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 13 / 34

slide-18
SLIDE 18

Environment modeling: Depth image of the scene Depth image reconstruction from base geometry

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 13 / 34

slide-19
SLIDE 19

Illumination Setup: Light-depth environment map We combine panorama and depth image to obtain the Light-depth environment map

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 14 / 34

slide-20
SLIDE 20

Illumination Setup: Light-depth environment map We combine panorama and depth image to obtain the Light-depth environment map

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 14 / 34

slide-21
SLIDE 21

Modeling new synthetic elements Modeled elements (in blue) placed in the scene interact with the environment

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 15 / 34

slide-22
SLIDE 22

Integration and rendering Augmented scene being rendered by our algorithm implemented in ARLuxrender

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 16 / 34

slide-23
SLIDE 23

Integration and rendering Final scene after rendering process

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 16 / 34

slide-24
SLIDE 24

RENDERING PROCESS

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 17 / 34

slide-25
SLIDE 25

Primitives classification

1

Synthetic primitives: the objects that are new to the scene. They don’t exist in the original environment.

2

Support primitives: surfaces present in the original environment that needs to receive shadows and reflections from the synthetic primitives.

3

Environment primitives: all the surfaces of the original environment that need to be take into account for the reflection and shadow computation for the other primitive types.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 18 / 34

slide-26
SLIDE 26

Primitives classification Scene primitives

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 19 / 34

slide-27
SLIDE 27

Primitives classification Scene primitives

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 19 / 34

slide-28
SLIDE 28

Shadows

C a m OE p Environment Map

E n v i r

  • n

m e n t p r i m i t i v e support primitive synthetic primitive r(p, L TW(wi))

li wi

r(p, li-p) Directional approach: The ray r(p, LTW(ωi)) is not occluded, so the light li illuminates the point p.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 20 / 34

slide-29
SLIDE 29

Shadows

C a m OE p Environment Map

E n v i r

  • n

m e n t p r i m i t i v e support primitive synthetic primitive r(p, L TW(wi))

li wi

r(p, li-p) Directional approach: The ray r(p, LTW(ωi)) is not occluded, so the light li illuminates the point p. Light-positional approach: The ray r(p, li − p) is occluded by a synthetic object, thus it gets the correct world-based estimation.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 20 / 34

slide-30
SLIDE 30

Shadows

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 21 / 34

slide-31
SLIDE 31

Shadows Realistic shadows effects. Shadows are consistent with world light positions.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 22 / 34

slide-32
SLIDE 32

Reflections

C a m p q

np

Environment Map

E n v i r

  • n

m e n t p r i m i t i v e support primitive synthetic primitive

wi r ( p , w

i

) OE qL

Directional approach: The reflection color value is given by the ray ωi.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 23 / 34

slide-33
SLIDE 33

Reflections

C a m p q

np

Environment Map

E n v i r

  • n

m e n t p r i m i t i v e support primitive synthetic primitive

wi r ( p , w

i

) OE qL

Directional approach: The reflection color value is given by the ray ωi. Light-positional approach: The ray r(p, ωi) intersects the environment mesh at point q. The radiance of q is stored at qL direction in the light-depth map. The reflection value is given by qL.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 23 / 34

slide-34
SLIDE 34

Reflections

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 24 / 34

slide-35
SLIDE 35

Reflections Reflections using the directional approach

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 25 / 34

slide-36
SLIDE 36

Reflections Reflections using the light-positional approach

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 25 / 34

slide-37
SLIDE 37

RESULTS AND CONCLUSIONS

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 26 / 34

slide-38
SLIDE 38

Results Full panoramic scene in Latitude-longitude format.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 27 / 34

slide-39
SLIDE 39

Results Fisheye lens view of the panoramic image.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 28 / 34

slide-40
SLIDE 40

Results Rendering result using a perspective camera.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 29 / 34

slide-41
SLIDE 41

Results Camera position out from the environment capture position.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 30 / 34

slide-42
SLIDE 42

Results Couch with two balls. The balls apply a soft deformation on the couch mesh and texture.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 31 / 34

slide-43
SLIDE 43

Conclusions

1

The Light-positional approach gives the correct world based accounts for rendering.

2

One pass rendering solution. We dont need to make compositions

  • r work with layers.

3

Extensible to Cycles (GPU render of Blender) render or other renderers.

4

Light-depth map idea can be used for real time rendering in other contexts (games, interactive visualization, etc.)

5

We can handle camera rotations and translations!

6

It’s possible apply deformation of mesh and texture deformation to the modeled environment.

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 32 / 34

slide-44
SLIDE 44

Links of the project and related softwares

1

http://www.impa.br/˜zang/arlux/

2

http://w3.impa.br/˜zang/blenderconf/

3

http://www.luxrender.net/en_GB/index

4

http://www.blender.org/

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 33 / 34

slide-45
SLIDE 45

Questions?

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 34 / 34

slide-46
SLIDE 46

Questions? THANKS!

  • D. Felinto, A. Zang, L. Velho

(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 34 / 34