Production framework for full panoramic scenes with photo-realistic - - PowerPoint PPT Presentation
INSTITUTE OF PURE AND APPLIED MATHEMATICS Production framework for full panoramic scenes with photo-realistic augmented reality Dalai Felinto , Aldo Zang and Luiz Velho Fisheries Centre, UBC - Vancouver, Canada Visgraf
INSTITUTE OF PURE AND APPLIED MATHEMATICS
Dalai Felinto⋆, Aldo Zang† and Luiz Velho†
⋆ Fisheries Centre, UBC - Vancouver, Canada † Visgraf Laboratory, IMPA - Rio de Janeiro, Brazil
October 2012
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 1 / 34
Outline
1
Part 1 Introduction Technique pipeline
Environment capture
2
Part 2 Rendering process
Scene primitives Shadows and reflections accounts
3
Part 3 Results Conclusions
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 2 / 34
Introduction From a full panoramic HDR image we reconstruct the real scene, insert and illuminate synthetic elements in a photorealistic way using accurate shadows and reflections.
Real Panorama → Processing → Augmented Panorama
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 3 / 34
Technique pipeline
1
Environment capture and calibration
2
Scene modeling
3
Scene depth image computation
4
Illumination setup
5
Synthetic elements
6
Integration and rendering
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 4 / 34
Environment Capture Space partition for Nikon 10.5mm: 5 photos around z- axis, 1 zenith photo and 1 nadir photo. Capture equipment: Manfrotto Pano Head, Nikon DX2S camera and Nikon 10.5mm fisheye lens.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 5 / 34
Panorama Stitching Stitching of multiple images Stitched and blended image
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 6 / 34
Panorama Calibration
Advantages of a calibration system:
1
Allow to move important sampling regions off the poles.
2
Less concern on tripod alignment for picture capturing.
3
It works with panoramas obtained from the internet.
4
Optimal aligned axis for the world reconstruction.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 7 / 34
Panorama Calibration
Calibration system
The four points on floor determines the orientations axis for the
(horizon), xz and yz slice planes.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 8 / 34
Panorama Calibration
Calibration account
Orientation: x = ( p0 × p1) × ( p3 × p2) y = ( p1 × p2) × ( p0 × p3) z = x × y y = z × x We need, however, to gather more data - as the orientation alone is not sufficient!
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 9 / 34
Panorama Calibration
Calibration account
Orientation: x = ( p0 × p1) × ( p3 × p2) y = ( p1 × p2) × ( p0 × p3) z = x × y y = z × x We need, however, to gather more data - as the orientation alone is not sufficient! Any arbitrary positive non-zero value for the camera height will produce a different (scaled) reconstruction of the original geometry in the 3d world. We have a dual system with the rectangle dimensions and the camera
available.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 9 / 34
Illumination Setup: HDR environment map EV = 0 EV = -2 EV = -4 EV = -6 Full panoramic image in HDR format.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 10 / 34
Environment modeling
1
Used to compute scene depth, for visibility and shadows
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 11 / 34
Environment modeling
1
Used to compute scene depth, for visibility and shadows
2
Provides the light position in world space for the reflection rays
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 11 / 34
Environment modeling
1
Used to compute scene depth, for visibility and shadows
2
Provides the light position in world space for the reflection rays
3
The modeled environment serves as support surfaces for shadows and reflections rendering of synthetic elements
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 11 / 34
Environment modeling Modeled base geometry (wireframe)
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 12 / 34
Environment modeling Modeled base geometry (polygons)
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 12 / 34
Environment modeling: Depth image of the scene Depth image reconstruction from base geometry
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 13 / 34
Environment modeling: Depth image of the scene Depth image reconstruction from base geometry
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 13 / 34
Illumination Setup: Light-depth environment map We combine panorama and depth image to obtain the Light-depth environment map
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 14 / 34
Illumination Setup: Light-depth environment map We combine panorama and depth image to obtain the Light-depth environment map
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 14 / 34
Modeling new synthetic elements Modeled elements (in blue) placed in the scene interact with the environment
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 15 / 34
Integration and rendering Augmented scene being rendered by our algorithm implemented in ARLuxrender
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 16 / 34
Integration and rendering Final scene after rendering process
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 16 / 34
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 17 / 34
Primitives classification
1
Synthetic primitives: the objects that are new to the scene. They don’t exist in the original environment.
2
Support primitives: surfaces present in the original environment that needs to receive shadows and reflections from the synthetic primitives.
3
Environment primitives: all the surfaces of the original environment that need to be take into account for the reflection and shadow computation for the other primitive types.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 18 / 34
Primitives classification Scene primitives
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 19 / 34
Primitives classification Scene primitives
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 19 / 34
Shadows
C a m OE p Environment Map
E n v i r
m e n t p r i m i t i v e support primitive synthetic primitive r(p, L TW(wi))
li wi
r(p, li-p) Directional approach: The ray r(p, LTW(ωi)) is not occluded, so the light li illuminates the point p.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 20 / 34
Shadows
C a m OE p Environment Map
E n v i r
m e n t p r i m i t i v e support primitive synthetic primitive r(p, L TW(wi))
li wi
r(p, li-p) Directional approach: The ray r(p, LTW(ωi)) is not occluded, so the light li illuminates the point p. Light-positional approach: The ray r(p, li − p) is occluded by a synthetic object, thus it gets the correct world-based estimation.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 20 / 34
Shadows
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 21 / 34
Shadows Realistic shadows effects. Shadows are consistent with world light positions.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 22 / 34
Reflections
C a m p q
np
Environment Map
E n v i r
m e n t p r i m i t i v e support primitive synthetic primitive
wi r ( p , w
i
) OE qL
Directional approach: The reflection color value is given by the ray ωi.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 23 / 34
Reflections
C a m p q
np
Environment Map
E n v i r
m e n t p r i m i t i v e support primitive synthetic primitive
wi r ( p , w
i
) OE qL
Directional approach: The reflection color value is given by the ray ωi. Light-positional approach: The ray r(p, ωi) intersects the environment mesh at point q. The radiance of q is stored at qL direction in the light-depth map. The reflection value is given by qL.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 23 / 34
Reflections
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 24 / 34
Reflections Reflections using the directional approach
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 25 / 34
Reflections Reflections using the light-positional approach
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 25 / 34
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 26 / 34
Results Full panoramic scene in Latitude-longitude format.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 27 / 34
Results Fisheye lens view of the panoramic image.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 28 / 34
Results Rendering result using a perspective camera.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 29 / 34
Results Camera position out from the environment capture position.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 30 / 34
Results Couch with two balls. The balls apply a soft deformation on the couch mesh and texture.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 31 / 34
Conclusions
1
The Light-positional approach gives the correct world based accounts for rendering.
2
One pass rendering solution. We dont need to make compositions
3
Extensible to Cycles (GPU render of Blender) render or other renderers.
4
Light-depth map idea can be used for real time rendering in other contexts (games, interactive visualization, etc.)
5
We can handle camera rotations and translations!
6
It’s possible apply deformation of mesh and texture deformation to the modeled environment.
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 32 / 34
Links of the project and related softwares
1
http://www.impa.br/˜zang/arlux/
2
http://w3.impa.br/˜zang/blenderconf/
3
http://www.luxrender.net/en_GB/index
4
http://www.blender.org/
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 33 / 34
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 34 / 34
(IMPA) Production framework for full panoramic scenes with photo-realistic augmented reality 34 / 34