GradNet: Unsupervised Deep Screened Poisson Reconstruction for - PowerPoint PPT Presentation

GradNet: Unsupervised Deep Screened Poisson Reconstruction for Gradient‐Domain Rendering Jie Guo 1 Mengtian Li 1 Quewei Li 1 Yuting Qiang 1 Bingyang Hu 1 Yanwen Guo 1 Ling‐Qi Yan 2 1 State Key Lab for Novel Software 2 University of California, Santa Barbara Technology, Nanjing University sa2019.siggraph.org

Path tracing Light Mirror Diffuse Diffuse SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Path tracing error / 2 = samples * 4 SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Gradient‐domain Rendering Base Path Offset Path Light Mirror Diffuse Diffuse SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Gradient‐domain Rendering SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Gradient‐domain Rendering SA2019.SIGGRAPH.ORG 6 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Screened Poisson Reconstruction • For a base image and gradients rendered by any gradient‐ domain algorithms, we can reconstruct the final image by solving the following optimization problem SA2019.SIGGRAPH.ORG 7 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Screened Poisson Reconstruction • For a base image and gradients rendered by any gradient‐ domain algorithms, we can reconstruct the final image by solving the following optimization problem Data term Gradient term SA2019.SIGGRAPH.ORG 8 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Screened Poisson Reconstruction • Lp norm p = 2 means L 2 reconstruction p = 1 means L 1 reconstruction SA2019.SIGGRAPH.ORG 9 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Regularization • A regularized version of the screened Poisson solver can be written as Regularizer SA2019.SIGGRAPH.ORG 9 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Regularized Screen Poisson Reconstruction Rendering‐specific features SA2019.SIGGRAPH.ORG 1 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Our Idea Rendering‐specific features SA2019.SIGGRAPH.ORG 1 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Related Work METHOD DEEP LEARNING BASED AUXILIARY BUFFERS PERFORMANCE L1 × × ≈0.45s GPU CV [Rousselle et al. 2016] × × ≈2s CPU LTS [Ha et al. 2019] × × ≈1.7s CPU NFOR [Bitterli et al. 2016] × √ ≈200s CPU REG [Manzi et al. 2016] × √ ≈60s GPU KPCN [Bako et al. 2017] √ Supervised √ ≈1.7s GPU [Kettunen et al. 2019] √ Supervised √ ≈0.3s GPU SA2019.SIGGRAPH.ORG 4 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Related Work METHOD DEEP LEARNING BASED AUXILIARY BUFFERS PERFORMANCE L1 × × ≈0.45s GPU CV [Rousselle et al. 2016] × × ≈2s CPU LTS [Ha et al. 2019] × × ≈1.7s CPU NFOR [Bitterli et al. 2016] × √ ≈200s CPU REG [Manzi et al. 2016] × √ ≈60s GPU KPCN [Bako et al. 2017] √ Supervised √ ≈1.7s GPU [Kettunen et al. 2019] √ Supervised √ ≈0.3s GPU SA2019.SIGGRAPH.ORG 4 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Related Work (REG [Manzi et al. 2016]) Feature Bases via Truncated SVD. Solved by an iteratively reweighted least squares (IRLS) approach SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Related Work (KPCN) SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Related Work ([Kettunen et al. 2019]) 1. Using gradients as an additional feature 2. Adopting a new perceptual loss SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Our Method • Deep learning based • Unsupervised • Fast to reconstruct high‐quality image SA2019.SIGGRAPH.ORG 1 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Our Solution • Replace the traditional optimization in screened Poisson reconstruction with GradNet SA2019.SIGGRAPH.ORG 10 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Network Architecture • Multi‐branch auto‐encoder with dual skip connection Low‐frequency contents High‐frequency details SA2019.SIGGRAPH.ORG 20 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Network Architecture • Multi‐branch auto‐encoder with dual skip connection Data branch G‐branch for generating derivative Gradient branch SA2019.SIGGRAPH.ORG 21 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Impact of the Branches One‐branch encoder weaken the effects of sparse image gradients SA2019.SIGGRAPH.ORG 22 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Dynamic Range Compression • We employ the μ‐law transformation to compress HDR data • The μ‐law transformation makes the training process easier than naïve log transformation SA2019.SIGGRAPH.ORG 23 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Loss Function • The loss function contains 3 items • data item, gradient item and first‐order item • Data item: SA2019.SIGGRAPH.ORG 24 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Loss Function • The loss function contains 3 items • data item, gradient item and first‐order item • Gradient item: SA2019.SIGGRAPH.ORG 25 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Impact of the Gradient Loss I b I dx I dy w. gradient loss Reference w.o. gradient loss SA2019.SIGGRAPH.ORG 26 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Loss Function • The loss function contains 3 items • data item, gradient item and first‐order item • The first‐order item: SA2019.SIGGRAPH.ORG 27 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

First‐order Regularization • The first‐order regularization defines as follow � 1 � � � � 𝐻 � � �𝐺 � � 𝑥 �,� 𝐽 � � 𝐽 � � 𝐺 � � 𝑂 𝑂 � ��� �∈� � Derivative of I i Difference of Neighboring pixels with respect to F i auxiliary features around index i SA2019.SIGGRAPH.ORG 28 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

First‐order Regularization • The first‐order regularization encourages nearby pixels to lie on a hyper‐plane parameterized by G G SA2019.SIGGRAPH.ORG 29 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Impact of the First‐order Loss w.o. the first-order loss w. the first-order loss reference SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Bias of Network μ will introduce bias to the reconstructed image Reference (mean: 0.204) μ = 16 (mean: 0.187) μ = 128 (mean: 0.176) μ = 1024 (mean: 0.163) SA2019.SIGGRAPH.ORG CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Post‐Processing • We find a simple post‐processing step can reduce the bias Gaussian filter with r = 45 and σ = 15 Without P.P. With P.P. SA2019.SIGGRAPH.ORG 32 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Training Details • Adam optimizer • β1 = 0.5 and β2 = 0.999 • Initial learning rate = 0.0001 and decays with the power of 0.95 for every other epoch • λ follows the schedule • Train main branches and G‐branch alternatively for 50 epochs with 32 mini‐batches SA2019.SIGGRAPH.ORG 33 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Dataset • Randomly perturb 9 base scenes and render to 900 high‐ resolution images with 64 spp SA2019.SIGGRAPH.ORG 34 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Dataset • 12654 patches with 256x256 resolution are extracted from these 900 high‐resolution images SA2019.SIGGRAPH.ORG 35 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA

Results SA2019.SIGGRAPH.ORG 36 CONFERENCE 17‐20 November 2019 ‐ EXHIBITION 18‐20 November 2019 ‐ BCEC , Brisbane, AUSTRALIA