Local Shape Editing at the Compositing Stage Carlos J. Zubiaga, Gaël Guennebaud, Romain Vergne, Pascal Barla
Compositing Shading Buffers Diffuse Reflection Final Image ∑ Transparent Subsurface Emission
Compositing Auxiliary Buffers Shading Buffers Diffuse Normals Reflection Depth/Position Diffuse Coeff. Transparent Subsurface Object ID Emission Ambient Occ.
Compositing Effects Shading Buffers Auxiliary Buffers Diffuse Reflection Normals Depth/Position Final Image Relighting DOF / Fog
Compositing Effects Limitations Shading Buffers Auxiliary Buffers Diffuse Reflection Normals Depth/Position Shape modifications do not change shading Lighting is lost in the rendering process Costly re-rendering is needed
Goal Grant real-time local al sha hape modification in post-processing in a plau ausibl ible way
Previous Work
Lighting Reconstruction A Signal-Processing Framework for Inverse Rendering [Ramammorthi et al 01] Inv nverse Rend ndering ng is an an ill ll-Posed Proble lem • Photograph Render Reflectance and Natural Illumination from a Single Image [Lombardi et al 12] Assume na natural al li lighting ng an and lo low e ent ntropy • Stat atisti stical cal BRDFs an and a l a low d detai ail l li lighting ng • Photograph Reconstructed Illumination Env. Light Map Decoupling of lighting and material is not ot nece cessa sary y to manipulate appearance
Appearance Manipulation I Image Based Material Editing [Khan et al 06] Estimates ates of 3 3D shap ape & • li lighting ng for re-rend nderi ring ng Input HDR Modification 1 Modification 2 Control Control Interactive Reflection Editing [Ritschel et al 09] Manipulate reflections on 3D • away from physical restrictions Input Input Result Result Manipulation performed in in fu full 3D: not adapted to compositing
Appearance Manipulation II Surface flows for image-based shading design [Vergne et al 12] Use depth an and no normal al b buffer ers • to deform/warp arp imag ages Sahding effects Texture effects MatCap Decomposition for Dynamic Appearance Manipulation [Zubiaga et al 15] Modify lighting and material appearance from • MatCaps Avoid decoupling of lighting and material • Add reflections Shinier material A novel approach to manipulate exi xisti ting appearance in co complex ex renderings is required
Main Idea Reconstruct Pre-filtered Environment Maps (PEM) per object/material
Frequency-dependent Reconstruction Approach Diffuse Low-Frequency • Spherical Harmonics • Reflections High-Frequency • Detailed PEM •
Reconstruction
Diffuse Reconstruction Reconstruction Input +1 Spherical Harmonics Diffuse Shading Quadratic Programming (LS + “ ≤ ” ) Fitting 0 Diffuse Diffuse PEM Gauss Map Residuals Residuals corresponds to darkening by occlusion reconstructed Residuals can be reintroduced -1 Mean Least Square
Reflection Reconstruction High Resolution Low Resolution Blurring Weighting • Interpolation Favorize smallest quads Discontinuities Spherical barycentric coordinates •
Reflection Reconstruction 1st Hole Filling Input Harmonic interpolation on tessellated sphere 2nd Regularize Sligthly blur discontinues Front Back of disconnected polygons Dual Paraboloid Map
Recontruction Validation Diffuse Reflections Diff × 100 Diff × 5 Input Ground Truth Render Recontructed Render Recontructed Input Ground Truth Diff × 5 Diff × 10
Reconstruction Validation Sphere in Diffuse Head Vase perspective Shading Shading Shading Normals Normals Normals Reflection
Re-compositing
General Pipeline Recompositing Reconstruction Combine original & PEM Diffuse Modified Original Occlusion Recomposited PEM Reflection
Diffuse pipeline Recomposited Reconstructed Residual Re-introduce • Occlusion local darkening Linear interpolation Preserved occlusion • PEM Normal Original
Reflections pipeline Recomposited Original Diff Normals Occlusion Binary Mask • Disable modification • Linear interpolation Avoid ghosting effects • PEM Orig. Mod. Reconstructed
Recap Diffuse Reflections Add residuals Interpolation Interpolate using occclusion
Results
Normal Modification Comparison Ours Ours Ground Truth Ground Truth
Complex Scene Reconstruction
Timings (ms) Model Diffuse Reflection Reconstruction Reconstruction Partial Reconstruction Hole filling Red Sphere 110 315 215 Red Head 184 470 280 Red Vase 147 385 192 Cup 29 45 325 Kettle 65 97 650 Black Vase 64 133 4000 Truck Body 119 120 6000 Front Mudguard 25 45 2500 Rear Mudguard 14 26 3700
Conclusions Pl Plaus usib ible le pr pre-fi filt ltere ered d env nv. . ma maps ps are re su suffi ffici cient nt to mo modif ify y lo loca cal l sh shap ape at at the the co compo mposi siting ting st stag age Limitations Future Work Restricted to opaque objects Extension to spatially-varying reflectance Limited by the geometric complexity Output more info at the rendering stage Normal mapping, not displacement Combine with manipulation of materials
Thank you http://prism-network.eu/ https://www.thefoundry.co.uk/ http://gratin.gforge.inria.fr/
Complex Scenes Input Normal Specular Diffuse Shading Shading Surface ID Ambient Specular Occlusion Occlusion
Recommend
More recommend
