Local Shape Editing at the Compositing Stage Carlos J. Zubiaga, Gal - - PowerPoint PPT Presentation

Local Shape Editing at the Compositing Stage

Carlos J. Zubiaga, Gaël Guennebaud, Romain Vergne, Pascal Barla

Compositing

Final Image Subsurface Emission Transparent

∑

Diffuse Reflection

Shading Buffers

Subsurface Emission Transparent Diffuse Reflection

Shading Buffers

Compositing

Diffuse Coeff. Ambient Occ.

Auxiliary Buffers

Object ID Normals Depth/Position

Compositing Effects

DOF / Fog Relighting Final Image

Auxiliary Buffers Shading Buffers

Normals Depth/Position Diffuse Reflection

Compositing Effects Limitations

Normals

Auxiliary Buffers

Depth/Position

Shading Buffers

Diffuse Reflection

Shape modifications do not change shading Lighting is lost in the rendering process Costly re-rendering is needed

Goal

Grant real-time local al sha hape modification in post-processing in a plau ausibl ible way

Previous Work

Lighting Reconstruction

Reflectance and Natural Illumination from a Single Image [Lombardi et al 12]

• Assume na

natural al li lighting ng an and lo low e ent ntropy

• Stat

atisti stical cal BRDFs an and a l a low d detai ail l li lighting ng A Signal-Processing Framework for Inverse Rendering [Ramammorthi et al 01]

• Inv

nverse Rend ndering ng is an an ill ll-Posed Proble lem

Photograph Render

• Env. Light Map

Photograph Reconstructed Illumination

Decoupling of lighting and material is not

• t nece

cessa sary y to manipulate appearance

Appearance Manipulation I

Image Based Material Editing [Khan et al 06]

• Estimates

ates of 3 3D shap ape & li lighting ng for re-rend nderi ring ng Interactive Reflection Editing [Ritschel et al 09]

• Manipulate reflections on 3D

away from physical restrictions

Input HDR Modification 1 Modification 2 Input Input Result Result Control Control

Manipulation performed in in fu full 3D: not adapted to compositing

Appearance Manipulation II

Surface flows for image-based shading design [Vergne et al 12]

• Use depth an

and no normal al b buffer ers to deform/warp arp imag ages MatCap Decomposition for Dynamic Appearance Manipulation [Zubiaga et al 15]

• Modify lighting and material appearance from

MatCaps

• Avoid decoupling of lighting and material

Texture effects

Add reflections Shinier material

Sahding effects

A novel approach to manipulate exi xisti ting appearance in co complex ex renderings is required

Main Idea

Reconstruct Pre-filtered Environment Maps (PEM) per object/material

Frequency-dependent Reconstruction Approach

Diffuse

• Low-Frequency
• Spherical Harmonics

Reflections

• High-Frequency
• Detailed PEM

Reconstruction

Diffuse Reconstruction

Input Diffuse Shading Gauss Map Diffuse PEM Diffuse reconstructed Residuals Reconstruction Spherical Harmonics Fitting Quadratic Programming (LS + “≤”) Mean Least Square +1

• 1

Residuals corresponds to darkening by occlusion Residuals can be reintroduced

Reflection Reconstruction

High Resolution Low Resolution

Interpolation Spherical barycentric coordinates Weighting Favorize smallest quads

• Blurring
• Discontinuities

Reflection Reconstruction

Dual Paraboloid Map Front Back 1st Hole Filling 2nd Regularize

Harmonic interpolation

• n tessellated sphere

Input

Sligthly blur discontinues

• f disconnected polygons

Recontruction Validation

Diffuse

Diff ×5 Render Recontructed Input Ground Truth Diff ×5 Diff ×100 Diff ×10

Reflections

Render Recontructed Input Ground Truth

Reconstruction Validation

Normals Shading Diffuse Reflection Sphere in perspective Head Normals Shading Vase Normals Shading

Re-compositing

General Pipeline

Original PEM Occlusion Diffuse Reflection Recomposited Modified Recompositing Combine original & PEM Reconstruction

Diffuse pipeline

Normal Reconstructed Occlusion PEM Linear interpolation

• Preserved occlusion

Recomposited Residual

• Re-introduce

local darkening Original

Reflections pipeline

Reconstructed Occlusion

• Binary Mask
• Disable modification

Recomposited Diff Normals Linear interpolation

• Avoid ghosting effects

Orig. PEM Mod. Original

Recap

Interpolate using occclusion Interpolation

Diffuse

Add residuals

Reflections

Results

Normal Modification Comparison

Ours Ground Truth Ground Truth Ours

Complex Scene Reconstruction

Timings (ms)

Model Diffuse Reconstruction Reflection Reconstruction Partial Reconstruction Hole filling Red Sphere 110 315 215 Red Head 184 470 280 Red Vase 147 385 192 Cup 29 45 325 Kettle 65 97 650 Black Vase 64 133 4000 Truck Body 119 120 6000 Front Mudguard 25 45 2500 Rear Mudguard 14 26 3700

Conclusions

Restricted to opaque objects Limited by the geometric complexity Normal mapping, not displacement

Pl Plaus usib ible le pr pre-fi filt ltere ered d env nv. . ma maps ps are re su suffi ffici cient nt to mo modif ify y lo loca cal l sh shap ape at at the the co compo mposi siting ting st stag age Limitations

Extension to spatially-varying reflectance Output more info at the rendering stage Combine with manipulation of materials

Future Work

Thank you

http://prism-network.eu/ http://gratin.gforge.inria.fr/ https://www.thefoundry.co.uk/

Complex Scenes Input

Normal Specular Shading Surface ID Ambient Occlusion Specular Occlusion Diffuse Shading