The Virtual Photo Set (VPS) IIS11-0081: Data-driven scene - - PowerPoint PPT Presentation

▶
the virtual photo set vps
SMART_READER_LITE
LIVE PREVIEW

The Virtual Photo Set (VPS) IIS11-0081: Data-driven scene - - PowerPoint PPT Presentation

Dec 22, 2022 124 likes •588 views

The Virtual Photo Set (VPS) IIS11-0081: Data-driven scene characterization for realistic rendering 2017-11-14 Project overview Project title: IIS11-0081 Data-driven scene characterisation for realistic rendering Project leader: Anders

slide-1
SLIDE 1

The Virtual Photo Set (VPS)

IIS11-0081: Data-driven scene characterization for realistic rendering

2017-11-14

slide-2
SLIDE 2

Project overview

Project title: IIS11-0081 Data-driven scene characterisation for realistic rendering Project leader: Anders Ynnerman and Jonas Unger Principal investigators: Michael Felsberg, Fredrik Gustafsson, Reiner Lenz, Jonas Unger, Anders Ynnerman Funding amount: 27,000,000 SEK Funding period: 2012-01-01 - 2016-12-31

slide-3
SLIDE 3

The image paradigm shift

The vision of this project is to develop the foundation for the next generation imaging pipelines where reality can be edited and virtual and real objects can be seamlessly mixed.

slide-4
SLIDE 4

Application drivers

  • Digital design
  • Product visualization
  • Special effects in movies
  • Computer games
  • Augmented reality
  • ...
slide-5
SLIDE 5

Major challenges

A High Dynamic Range (HDR) imaging pipeline

  • Multi-modal input data
  • Tracking of input devices
  • TBs of data per capture
  • On-line user feedback

Processing and editing Rendering Scene Capture

  • Geometry and light source

extraction, and scene editing

  • Data representations and

compression

  • Interactive processing and

user feedback

  • Ultra-realistic off-line

rendering

  • Efficient material models
  • Compression for real-time

applications

  • Tone mapping
slide-6
SLIDE 6

Project overview

Project team: C-Research, ITN, LiU, (Prof. Anders Ynnerman) Computer Vision Laboratory, ISY, LiU, (Prof. Michael Felsberg) Sensor Fusion Group, ISY, LiU, (Prof. Fredrik Gustafsson) 6 senior researchers, 6 PhD students, and 4 research engineers

slide-7
SLIDE 7

Collaborations

  • Linköping University
  • University of Southern California
  • Warwick University
  • Bangor University
  • IKEA Communications
  • SpheronVR AG
  • IrysTech
  • Swiss International
  • Volvo PVH

Scientific collaborations Industry collaborations New projects, funding proposals, and joint papers with new both academic and industrial partners

slide-8
SLIDE 8

Project results

Scientific output

  • 13 journal papers
  • 39 contributions at leading conferences in the field
  • 3 book chapters
  • 2 PhD theses (supported by this project) + 2 theses spring 2018
  • 1 Licentiate thesis
  • 5 M.Sc. theses

Open data and software

  • HDR-video sequence data sets for research and educational use
  • Lighting environments captured using HDR imaging
  • Depends workflow management system
  • LumaHDRv high dynamic range (HDR) video codec

URLs: [ www.hdrv.org ], [ www.dependsworkflow.net ], [ www.lumahdrv.org ]

Spin-offs

  • Materialeyes AB (measurement systems, methods, and 


algorithms for appearance capture )

  • MassVis AB
slide-9
SLIDE 9

VPS solutions driving the state-of-the-art

slide-10
SLIDE 10

Image based lighting

Current state-of-the-art: Lighting is captured as a still image at a single position and at a single instant in time

slide-11
SLIDE 11

HDR Light Probes

Virtual scene Virtual camera HDR environment map

L(x, ~ !o) = Z

Ω

Lenv(~ !i)⇢(x, ~ !i → ~ !o)(−~ !i · ~ nx)V (~ !i)d~ !i

  • Lighting is described in the panoramic HDR image
  • Each pixel corresponds to the scene radiance

incident over the solid angle subtended by the pixel

Lighting Material Cosine falloff Visibility

slide-12
SLIDE 12

Mixing Virtual and Real

L(x, ~ !o) = Z

Ω

Lenv(~ !i)⇢(x, ~ !i → ~ !o)(−~ !i · ~ nx)V (~ !i)d~ !i

Lighting Material Cosine falloff Visibility

slide-13
SLIDE 13
slide-14
SLIDE 14

Requires high dynamic range (HDR) video and light re-projection onto scene geometry

Rendering using accurately measured lighting recovered using the VPS approach Image synthesis using previous state-of-the-art methods

Challenges (and VPS solutions)

Capture: lighting conditions, scene geometry, and material information

Light Fields

slide-15
SLIDE 15

Capture: lighting conditions, scene geometry, and material information

  • Robust capture of lighting conditions

in complex environments

  • High resolution HDR-video beyond

full HD

  • Effective dynamic range at least

1:10,000,000

  • Accurate radiometric calibration

LiU HDRv: http://www.hdrv.org Our demands on HDR-video go far beyond commercial solutions

Need for HDR Video

Challenges

slide-16
SLIDE 16 Affjne Transform Affjne Transform f(x,y) PSF Sampling (R,G,B) + d(x,y) p(x,y) Affjne Transform Sampling (R,G,B) + ND Filter ND Filter Sampling (R,G,B) + ND Filter F T1 T2 T3 I2 I3 I1 Zj

Capture: lighting conditions, scene geometry, and material information

  • Large scale data: ~1.5GB/s of floating point

pixel data

  • Minutes of capture leads to TBs of data
  • Real-time user feedback and HDR image 


reconstruction is a requirement

  • VPS result: Novel image reconstruction 


framework for multi-sensor systems suitable for 
 parallel computations and GPU implementation

  • Active in COST Action IC1005: Capture, storage, 


transmission and display of real-world lighting

HDR capture device

Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Anders Ynnerman, Jonas Unger, "A unified framework for multi-sensor HDR video reconstruction", Signal Processing : Image Communications, 29(2): 203-215, 2014.

  • J. Kronander, S. Gustavson, G. Bonnet, A.

Ynnerman, J. Unger: Unified HDR Reconstruction from RAW CFA Data, In proceedings of the IEEE International Conference on Computational Photography (ICCP), 2013

slide-17
SLIDE 17

Light Reprojection

  • Project HDR light fields onto 3D geometry
  • Efficient representation for data compression
  • Encodes high frequency features
  • User level editing of geometry, materials and light sources

Capture: lighting conditions, scene geometry, and material information

slide-18
SLIDE 18

Light Reprojection

  • Project HDR light fields onto 3D geometry
  • Efficient representation for data compression
  • Encodes high frequency features
  • User level editing of geometry, materials and light sources

Capture: lighting conditions, scene geometry, and material information

slide-19
SLIDE 19

Capture: lighting conditions, scene geometry, and material information Our demands on the model poses research challenges in tracking and geometry estimation

Scene reconstruction

  • Millimeter accuracy in recovered model
  • Submillimeter accuracy and sub degree

accuracy in camera pose and trajectory estimation

  • Fusion of image information and range data
  • Robustness to non-stationary objects in the

scene Challenges

slide-20
SLIDE 20

Light source extraction

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings

  • f EUSIPCO '13, 2013.
slide-21
SLIDE 21

Image synthesis

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings

  • f EUSIPCO '13, 2013.
slide-22
SLIDE 22

Captured Scene

slide-23
SLIDE 23

Tables removed

slide-24
SLIDE 24

Project HDR data

slide-25
SLIDE 25

Panoramic image displaying layout of the scene Recovered model

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings

  • f EUSIPCO '13, 2013.
slide-26
SLIDE 26

Room populated with virtual furniture

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings

  • f EUSIPCO '13, 2013.
slide-27
SLIDE 27

Norrköping

Computer graphics image

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings

  • f EUSIPCO '13, 2013.
slide-28
SLIDE 28

Norrköping

  • J. Unger, S. Gustavson, J. Kronander, P

. Larsson, G. Bonnet, and G. Kaiser. 2011. Next generation image based lighting using HDR video. In ACM SIGGRAPH 2011 Talks

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

Computer graphics image

slide-29
SLIDE 29

Norrköping

Computer graphics image

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, J. Löw, A. Ynnerman: Virtual Photo Sets - Spatially Varying Image Based Lighting using HDR- video, Computers and Graphics, Elsevier, Volume 37, Issue 7, 2013.

  • J. Unger, J. Kronander, P

. Larsson, S. Gustavson, A. Ynnerman, Temporally and Spatially Varying Image Based Lighting using HDR-video, Proceedings

  • f EUSIPCO '13, 2013.
slide-30
SLIDE 30

Surface light fields

Store the light field at surfaces in the scene Learning based compression using Exemplar Orthogonal Bases

Ehsan Miandji, Joel Kronander, Jonas Unger, "Learning Based Compression of Surface Light Fields for Real-time Rendering of Global Illumination Scenes", Proceedings of ACM SIGGRAPH ASIA 2013, 2013.

Real-time rendering Compression

slide-31
SLIDE 31

Compressed rendering

Ehsan Miandji, Joel Kronander, Jonas Unger, “Image reconstruction in reduced union of subspaces”, submitted to Eurographcis 2015

slide-32
SLIDE 32

Light field reconstruction

Ehsan Miandji, Joel Kronander, Jonas Unger. Compressive image reconstruction in reduced union of sub-

  • spaces. Computer Graphics Forum special issue Proceedings of Eurographics '15,
  • Vol. 34, No. 2, 2015
slide-33
SLIDE 33

Demonstrator integration

Depends workflow management system

  • Generality
  • Fusion of data from different input modalities
  • Modularity
  • The ability to reuse software components and customize the

processing to specific problems

  • Data provenance
  • A highly important aspect is to enable the ability to go back

and easily determine what generated certain results.

  • Performance
  • Large scale input data sizes (often in the order of TBs) and

the computational complexity of the processing algorithms present significant challenges.

  • Interaction
  • Advanced visualization and artistic interaction to steer the

computations.

Andrew Gardner, Jonas Unger, "Depends: Workflow Management Software for Visual Effects Production", Digital Production Symposium, DigiPro '14, Vancouver, Canada, August, 2014, DigiPro '14, 2014.

slide-34
SLIDE 34

A Swedish House

slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38

Library C++ API for encoding and decoding of HDR video Applications Encoding of HDR video frames Decoding of HDR video Playback of HDR video Dependencies Matroska media container VP9 (libvpx) from Google

Gabriel Eilertsen, Jonas Unger, and Rafal K. Mantiuk. Luma HDRv: an open source high dynamic range video codec optimized by large-scale testing. Accepted for SIGGRAPH '16 Talks, Annaheim, USA, 2016 Gabriel Eilertsen, Rafal K. Mantiuk, Jonas Unger. A High Dynamic Range Video Codec Optimized by Large Scale Testing, IEEE International Conference on Image Processing '16, Phoenix, USA, September, 2016.

HDR video compression

http://www.lumahdrv.org

slide-39
SLIDE 39

Real-time noise aware TMO

Display: adapt input video to display characteristics, viewing environment and perception

HDR

Edge-stopping spatial fjlter (fast detail extraction difgusion) Local tone curves (noise aware minimum contrast distortion) Input HDR-video Parameters: Peak luminance, Dynamic range, and Ambient light measurements Noise-aware control over image details Parameters: Local/global, Tone compression, Exposure, and Noise visibility control Parameters: Edge stop Parameter: Detail scaling, Noise visibility control Tone-mapped

  • utput

Data fmow User parameter Display parameter Noise estimate Noise model Detail layer, Base layer, Input frame, Inverse display model Real-time noise aware tone mapping operator

Gabriel Eilertsen, Rafal Mantiuk, and Jonas Unger, Real-time noise-aware tone mapping, Accepted for publication in ACM Transactions on Graphics proceedings of Siggraph Asia '15, Kobe, Japan November, 2015

slide-40
SLIDE 40
slide-41
SLIDE 41

Glasses free 3D display

Andrew Jones, Jonas Unger, Koki Nagano, Jay Busch, Xueming Yu, Hsuan- Yueh Peng, Oleg Alexander, Mark Bolas, Mark and Paul Debevec. An Automultiscopic Projector Array for Interactive Digital Humans. In ACM SIGGRAPH 2015 Emerging Technologies, August 2015. Andrew Jones, Jonas Unger, Jay Busch, Xueming Yu, Hsuan-Yueh Peng, Oleg Alexander and Paul Debevec. Creating a life-sized automultiscopic Morgan Spurlock for CNN's ``Inside Man''. In ACM SIGGRAPH 2014 Talks, Vancouver, Canada, August, 2014.

slide-42
SLIDE 42

Glasses free 3D display

335.28 CM 121.92 CM .63 DEGREES

216 PROJECTORS

SCREEN 385 CM 6 DEGREES CAMERAS

216 projectors 0.63 degrees

Projection screen 250 x 140 cm 340 cm

Projectors

slide-43
SLIDE 43

Glasses free 3D display

slide-44
SLIDE 44