[PPT] - Superv rvised Applications Niloy Mi Ni Mitra Ias asonas Kok PowerPoint Presentation

SLIDE 1

Ni Niloy Mi Mitra Ias asonas Kok

kkin

inos

s

Pau aul l Gu Guer errero Vl Vladim imir ir Ki Kim Kos

stas Rematas

Tob

bias Ri

Ritschel UCL UCL/Facebook UCL Adobe Research U Washington UCL

Deep Learning for Graphics

Superv rvised Applications

SLIDE 2

EG Course Deep Learning for Graphics

Timetable

2

Niloy Iasonas Paul Vova Kostas Tobias Introduction X X X X Theory X NN Basics X Supervised Applications X X X Data X Unsupervised Applications X Beyond 2D X X X Outlook X X X X X X

SLIDE 3

EG Course Deep Learning for Graphics

Fast (shared convolutions) Simple (dense)

FCN

Fully-Convolutional Network (FCN)

SLIDE 4

EG Course Deep Learning for Graphics

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. CVPR, 2015

FCN-based semantic segmentation

SLIDE 5

EG Course Deep Learning for Graphics

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. Yuille, Deeplab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, PAMI 2016

FCN-CRFs: Deeplab

SLIDE 6

EG Course Deep Learning for Graphics

Ground truth FCN FCN-DCRF

Deeplab v2 results

SLIDE 7

EG Course Deep Learning for Graphics

Deeplab v2 results

Ground truth FCN FCN-DCRF

SLIDE 8

EG Course Deep Learning for Graphics

Object Detection: Fast(er)-RCNN

Fast/Faster R-CNN

Good speed Good accuracy Intuitive Easy to use

Ross Girshick. “Fast R-CNN”. ICCV 2015. Shaoqing Ren, Kaiming He, Ross Girshick, & Jian Sun. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”. NIPS 2015.

SLIDE 9

EG Course Deep Learning for Graphics

Mask R-CNN

Mask R-CNN = Faster R-CNN with FCN on RoIs

Faster R-CNN FCN on RoI

SLIDE 10

EG Course “Deep Learning for Graphics”

Mask R-CNN results on COCO

SLIDE 11

EG Course “Deep Learning for Graphics”

1 keypoint = 1-hot “mask”
Human pose = 17 masks
Softmax over spatial locations
e.g. 562-way softmax on 56x56

Mask R-CNN for Human Keypoint Detection

SLIDE 12

EG Course Deep Learning for Graphics

Mask R-CNN frame-by-frame

SLIDE 13

EG Course “Deep Learning for Graphics”

Mask R-CNN frame-by-frame

SLIDE 14

EG Course Deep Learning for Graphics

I. Kokkinos, UberNet: Training a Universal CNN for Low- Mid- and High-Level Vision, CVPR 2017

https://github.com/jkokkin/UberNet

UberNet: a “universal” network for all tasks

SLIDE 15

EG Course Deep Learning for Graphics

“Inverse graphics”: understand how an image was generated from a scene If we focus on a single object category: surface-based models

UberNet: Universal Network DensePose: Unified model

What is the ultimate vision task?

SLIDE 16

EG Course Deep Learning for Graphics

R. A. Guler, G. Trigeorgis, E. Antonakos, P. Snape, S. Zafeiriou, I. Kokkinos,

DenseReg: Fully Convolutional Dense Shape Regression In-the-Wild, CVPR 2017

DenseReg: dense image-to-face regression

SLIDE 17

R. A. Guler, N. Neverova, I. Kokkinos “DensePose: Dense Human Pose Estimation In The Wild”, CVPR’18

DensePose-RCNN: ~25 FPS

DensePose: dense image-to-body correspondence

SLIDE 18

SLIDE 19

EG Course Deep Learning for Graphics

SFSNet: incorporating image formation in model

SfSNet: Learning Shape, Reflectance and Illuminance of Faces ‘in the wild' Soumyadip Sengupta Angjoo Kanazawa Carlos D. Castillo David W. Jacobs, CVPR 2018

SLIDE 20

EG Course Deep Learning for Graphics

Beyond single frames: end-to-end optical flow

SLIDE 21

EG Course Deep Learning for Graphics

End-to-end Structure From Motion

DeMoN: Depth and Motion Network for Learning Monocular Stereo, B. Ummenhofer, et al, CVPR 2017
Unsupervised learning of depth and ego-motion from video, T Zhou, M Brown, N Snavely, DG Lowe, CVPR 2017

SLIDE 22

EG Course Deep Learning for Graphics

Monocular depth & normal estimation

D. Eigen and R. Fergus, Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale

Convolutional Architecture, ICCV 2015

SLIDE 23

EG Course “Deep Learning for Graphics”

Graphics applications

SLIDE 24

EG Course Deep Learning for Graphics

Sketch Simplification

Learning to Simplify: Fully Convolutional Networks for Rough Sketch

Cleanup, Simon-Serra et al., 2016

Deep Extraction of Manga Structural Lines, Li et al., 2017

24

SLIDE 25

EG Course Deep Learning for Graphics

Sketch Simplification: Learning to Simplify fy

Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup, Simo-Serra et al. 25

SLIDE 26

EG Course Deep Learning for Graphics

Sketch Simplification: Learning to Simplify fy

Loss for thin edges saturates easily
Authors take extra steps to align input and ground truth edges

Pencil: input Red: ground truth

Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup, Simo-Serra et al. 26

SLIDE 27

EG Course Deep Learning for Graphics

Im Image Decomposition

A selection of methods:
Direct Instrinsics, Narihira et al., 2015
Learning Data-driven Reflectance Priors for Intrinsic Image Decomposition, Zhou et al.,

2015

Decomposing Single Images for Layered Photo Retouching, Innamorati et al. 2017

27

SLIDE 28

EG Course Deep Learning for Graphics

Im Image Decomposition: : Decomposing Sin ingle Im Images for Layered Photo Retouching

28

SLIDE 29

EG Course Deep Learning for Graphics

Colorization

Concurrent methods:
Let there be Color!, Iizuka et al., 2016
Colorful Image Colorization, Zhang et al. 2016
Learning Representations for Automatic Colorization, Larsson et al., 2016
Real-Time User-Guided Image Colorization with Learned Deep Priors, Zhang et al.

2017

29

SLIDE 30

EG Course Deep Learning for Graphics

Colorization: Let There Be Color!

Let there be Color!: Iizuka et al. 30

SLIDE 31

EG Course Deep Learning for Graphics

Colorization: Colorful Im Image Colorization

input direct regression probability distr.

utput

Image Credit: Colorful Image Colorization, Zhang et al. 31

SLIDE 32

EG Course Deep Learning for Graphics

Mesh Labeling / Segmentation

3D Mesh Labeling via Deep Convolutional Neural Networks, Guo et al. 2016 32

SLIDE 33

EG Course Deep Learning for Graphics

Mesh Labeling / Segmentation

3D Mesh Labeling via Deep Convolutional Neural Networks, Guo et al. 33

SLIDE 34

EG Course Deep Learning for Graphics

LDR to HDR Im Image Reconstruction:

Concurrently:
Deep Reverse Tone Mapping, Endo et al. 2017
HDR image reconstruction from a single exposure using deep CNNs,

Eilertsen et al. 2017

34

SLIDE 35

EG Course Deep Learning for Graphics

Reflectance Maps

Paint a sphere as if it is made
f a material under a certain

illumination of another

bject in a photo

Deep Reflectance Maps. Rematas et al. CVPR 2015 35

SLIDE 36

EG Course Deep Learning for Graphics

DeLight

Factor BRDF and (HDR) Illumination

Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. Georgoulis et al. PAMI 2017 36

SLIDE 37

EG Course Deep Learning for Graphics

3D volumes form Xrays

Single-Image Tomography: 3D Volumes from 2D Cranial X-Rays. Henzler et al. EG 2018

37

SLIDE 38

EG Course Deep Learning for Graphics

Deep Shading

Paint a z-buffer like a path tracer (AO, DOF, MB)

Deep Shading, Nalbach et al. EGSR 2017 38

SLIDE 39

EG Course Deep Learning for Graphics

Rendering Atmospherics

Speed up approx. 24 x Speed up approx. 24 x

39 Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks, Kallweit et al. SIGGRAPH Asia 2017

SLIDE 40

EG Course Deep Learning for Graphics

Rendering Atmospherics: RPNN

Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks, Kallweit et al. SIGGRAPH Asia 2017

In: Hierarchical representation of a cloud patch Out: incoming indirect radiance at patch center (incoming direct radiance is computed directly)

40

SLIDE 41

EG Course Deep Learning for Graphics

Denoising Renderings

Concurrent:
Kernel-Predicting Convolutional Networks

for Denoising Monte Carlo Renderings, Bako et al. 2017

Interactive Reconstruction of Monte Carlo

Image Sequences using a Recurrent Denoising Autoencoder, Chaitanya et al. 2017 (more on Autoencoders later)

Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings, Bako et al. 41

SLIDE 42

EG Course Deep Learning for Graphics

Denoising Renderings:

Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings, Bako et al. SIGGRAPH 2017 42

SLIDE 43

EG Course Deep Learning for Graphics

Geometry ry Abstraction / Simplification

Learning Shape Abstractions by Assembling Volumetric Primitives, Tulsiani et al. 2016 43

SLIDE 44

EG Course Deep Learning for Graphics

Geometry ry Abstraction / Simplification:

44 Learning Shape Abstractions by Assembling Volumetric Primitives, Tulsiani et al. 2016

SLIDE 45

EG Course Deep Learning for Graphics

Procedural Parameter Estimation

Interactive Sketching of Urban Procedural Models, Nishida et al. 2016 45

SLIDE 46

EG Course Deep Learning for Graphics

Procedural Parameter Estimation: In Interactive Sketching of f Urban Procedural Models

Interactive Sketching of Urban Procedural Models, Nishida et al. 46

SLIDE 47

EG Course Deep Learning for Graphics

Audio-driven facial animation

Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion, Karras et al. 2017

SLIDE 48

EG Course Deep Learning for Graphics

3D Pose Estimation: VNECT

VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera, Mehta et al., SIGGRAPH 2017

skeleton joint heatmap and 3d positions

50

SLIDE 49

EG Course Deep Learning for Graphics

Thank you!

http://geometry.cs.ucl.ac.uk/dl4g/

52