The Virtual Photo Set (VPS) IIS11-0081: Data-driven scene - PowerPoint PPT Presentation

The Virtual Photo Set (VPS) IIS11-0081: Data-driven scene characterization for realistic rendering 2017-11-14

Project overview Project title: IIS11-0081 Data-driven scene characterisation for realistic rendering Project leader: Anders Ynnerman and Jonas Unger Principal investigators: Michael Felsberg, Fredrik Gustafsson, Reiner Lenz, Jonas Unger, Anders Ynnerman Funding amount: 27,000,000 SEK Funding period: 2012-01-01 - 2016-12-31

The image paradigm shift The vision of this project is to develop the foundation for the next generation imaging pipelines where reality can be edited and virtual and real objects can be seamlessly mixed.

Application drivers • Digital design • Product visualization • Special effects in movies • Computer games • Augmented reality • ...

Major challenges A High Dynamic Range (HDR) imaging pipeline Scene Capture Processing and editing Rendering • Multi-modal input data • Geometry and light source • Ultra-realistic off-line • Tracking of input devices extraction, and scene editing rendering • TBs of data per capture • Data representations and • Efficient material models • On-line user feedback • Compression for real-time compression • Interactive processing and applications • Tone mapping user feedback

Project overview Project team: C-Research, ITN, LiU, (Prof. Anders Ynnerman) Computer Vision Laboratory, ISY, LiU, (Prof. Michael Felsberg) Sensor Fusion Group, ISY, LiU, (Prof. Fredrik Gustafsson) 6 senior researchers, 6 PhD students, and 4 research engineers

Collaborations New projects, funding proposals, and joint papers with new both academic and industrial partners Scientific collaborations Industry collaborations • Linköping University • IKEA Communications • University of Southern California • SpheronVR AG • Warwick University • IrysTech • Bangor University • Swiss International • Volvo PVH

Project results Scientific output 13 journal papers - 39 contributions at leading conferences in the field - 3 book chapters - 2 PhD theses (supported by this project) + 2 theses spring 2018 - 1 Licentiate thesis - 5 M.Sc. theses - Open data and software HDR-video sequence data sets for research and educational use - Lighting environments captured using HDR imaging - Depends workflow management system - LumaHDRv high dynamic range (HDR) video codec - URLs: [ www.hdrv.org ], [ www.dependsworkflow.net ], [ www.lumahdrv.org ] Spin-offs Materialeyes AB (measurement systems, methods, and 
 - algorithms for appearance capture ) MassVis AB -

VPS solutions driving the state-of-the-art

Image based lighting Current state-of-the-art: Lighting is captured as a still image at a single position and at a single instant in time

HDR Light Probes HDR environment map Virtual camera Virtual scene - Lighting is described in the panoramic HDR image - Each pixel corresponds to the scene radiance incident over the solid angle subtended by the pixel Z L ( x , ~ ! o ) = L env ( ~ ! i ) ⇢ ( x , ~ ! o )( − ~ n x ) V ( ~ ! i ) d ~ ! i → ~ ! i · ~ ! i Ω Lighting Material Cosine falloff Visibility

Mixing Virtual and Real Z L ( x , ~ ! o ) = L env ( ~ ! i ) ⇢ ( x , ~ ! o )( − ~ n x ) V ( ~ ! i ) d ~ ! i → ~ ! i · ~ ! i Ω Lighting Material Cosine falloff Visibility

Light Fields Challenges (and VPS solutions ) Capture: lighting conditions , scene geometry, and material information Rendering using accurately measured Image synthesis using previous lighting recovered using the VPS approach state-of-the-art methods Requires high dynamic range ( HDR ) video and light re-projection onto scene geometry

Need for HDR Video Capture: lighting conditions , scene geometry, and material information Challenges Robust capture of lighting conditions - in complex environments High resolution HDR-video beyond - full HD Effective dynamic range at least - 1:10,000,000 Accurate radiometric calibration - LiU HDRv: http://www.hdrv.org Our demands on HDR-video go far beyond commercial solutions

HDR capture device Capture: lighting conditions , scene geometry, and material information Large scale data: ~ 1.5GB/s of floating point - pixel data Minutes of capture leads to TBs of data - Real-time user feedback and HDR image 
 - reconstruction is a requirement VPS result : Novel image reconstruction 
 - framework for multi-sensor systems suitable for 
 parallel computations and GPU implementation Active in COST Action IC1005: Capture, storage, 
 - transmission and display of real-world lighting I 1 Sampling A ffj ne Transform ND Filter T 1 + (R,G,B) PSF d(x,y) f(x,y) p(x,y) I 2 Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Anders Ynnerman, T 2 ND Filter Sampling A ffj ne Transform + Z j (R,G,B) Jonas Unger, "A unified framework for multi-sensor HDR video I 3 T 3 reconstruction", Signal Processing : Image Communications, 29(2): A ffj ne Transform Sampling ND Filter + 203-215, 2014. (R,G,B) F J. Kronander, S. Gustavson, G. Bonnet, A. Ynnerman, J. Unger: Unified HDR Reconstruction from RAW CFA Data, In proceedings of the IEEE International Conference on Computational Photography (ICCP), 2013

Light Reprojection Capture: lighting conditions, scene geometry , and material information • Project HDR light fields onto 3D geometry • Efficient representation for data compression • Encodes high frequency features • User level editing of geometry, materials and light sources

Light Reprojection Capture: lighting conditions, scene geometry , and material information • Project HDR light fields onto 3D geometry • Efficient representation for data compression • Encodes high frequency features • User level editing of geometry, materials and light sources

Scene reconstruction Capture: lighting conditions, scene geometry , and material information Challenges Millimeter accuracy in recovered model - Submillimeter accuracy and sub degree - accuracy in camera pose and trajectory estimation Fusion of image information and range data - Robustness to non-stationary objects in the - scene Our demands on the model poses research challenges in tracking and geometry estimation

Light source extraction J. Unger, J. Kronander, P . Larsson, S. Gustavson, J. Löw, A. Ynnerman: J. Unger, J. Kronander, P . Larsson, S. Gustavson, A. Ynnerman, Temporally Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- and Spatially Varying Image Based Lighting using HDR-video, Proceedings video , Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013. of EUSIPCO '13, 2013.

Image synthesis J. Unger, J. Kronander, P . Larsson, S. Gustavson, J. Löw, A. Ynnerman: J. Unger, J. Kronander, P . Larsson, S. Gustavson, A. Ynnerman, Temporally Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- and Spatially Varying Image Based Lighting using HDR-video, Proceedings video , Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013. of EUSIPCO '13, 2013.

Captured Scene

Tables removed

Project HDR data

Panoramic image displaying layout of the scene Recovered model J. Unger, J. Kronander, P . Larsson, S. Gustavson, J. Löw, A. Ynnerman: J. Unger, J. Kronander, P . Larsson, S. Gustavson, A. Ynnerman, Temporally Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- and Spatially Varying Image Based Lighting using HDR-video, Proceedings video , Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013. of EUSIPCO '13, 2013.

Room populated with virtual furniture J. Unger, J. Kronander, P . Larsson, S. Gustavson, J. Löw, A. Ynnerman: J. Unger, J. Kronander, P . Larsson, S. Gustavson, A. Ynnerman, Temporally Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- and Spatially Varying Image Based Lighting using HDR-video, Proceedings video , Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013. of EUSIPCO '13, 2013.

Norrköping Computer graphics image J. Unger, J. Kronander, P . Larsson, S. Gustavson, J. Löw, A. Ynnerman: J. Unger, J. Kronander, P . Larsson, S. Gustavson, A. Ynnerman, Temporally Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- and Spatially Varying Image Based Lighting using HDR-video, Proceedings video , Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013. of EUSIPCO '13, 2013.

Norrköping Computer graphics image J. Unger, J. Kronander, P . Larsson, S. Gustavson, J. Löw, A. Ynnerman: J. Unger, S. Gustavson, J. Kronander, P . Larsson, G. Bonnet, Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- and G. Kaiser. 2011. Next generation image based lighting video , Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013. using HDR video. In ACM SIGGRAPH 2011 Talks

Norrköping Computer graphics image J. Unger, J. Kronander, P . Larsson, S. Gustavson, J. Löw, A. Ynnerman: J. Unger, J. Kronander, P . Larsson, S. Gustavson, A. Ynnerman, Temporally Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- and Spatially Varying Image Based Lighting using HDR-video, Proceedings video , Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013. of EUSIPCO '13, 2013.