Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 - PowerPoint PPT Presentation

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1

Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks 2012, Krizhevsky et.al. [source] 10/11 Going Deeper with Convolutions 2015, Szegedy et.al. [source] 17/11 Painting Style Transfer for Head Portraits using Convolutional Neural Networks 2016, Selim & Elgharib [source] + A Neural Algorithm of Artistic Style 2015, Gatys et.al. [source] + Image Style Transfer Using Convolutional Neural Networks 2016, Gatys et.al. [source] Convolutional Neural Networks 2

Painting Style Transfer for Head Portraits using Convolutional Neural Networks Selim & Elgharib, 2016 Convolutional Neural Networks 3

Outline ● Motivation ● Related work – Painting Transfer – Color Transfer – Extension to Video ● Painting trough CNNs ● Example Driven Spatial Constrains – Modified feature maps – Solving for the painting ● Results – Still images – Extension to video ● Discussion Convolutional Neural Networks 4

Motivation ● Usual style-transfer techniques often deform facial structures: no spatial constrains. → Human perception is very sensitive to changes in faces. ● Other strategies: – facial geometry (limited artistic styles: graphite & sketch painting) – image analogies (requires paired images: limited applicability) – color transfer: power maps (structure and color / no texture) Gatys, photo et.al. Selim, example et.al. Convolutional Neural Networks 5

Related: Painting Transfer ● Stroke based Example: Gatys et.al. – Simulates brush-stroke → Facial deformation!!! placement process – Facial landmarks as – Brush attributes: scale, guidance (AAM / ASM): orientation, color, opacity eyes, eyebrows, nose, ● Texture transfer mouth, face outline – Follow sample textures – Parts based techniques: – Texture synthesis non parametric sampling, ● Parts transfer MRF and spatial smoothness in – Parse the input image in parts reconstruction: graphite – Reconstruction from parts & sketch database Convolutional Neural Networks 6

Convolutional Neural Networks 7

Related Related: Color Transfer Related: Video ● Transfer color palette from ● Applying painting frame by example to input frame generates: flickering & shower-door. ● Not based in image ● Flickering: temporal analogies inconsistencies. ● Power maps: local statistics. → ● Shower-door: texture drifts Local distribution of light away from the object. ● Spatial constraints: warping ● Search better style, transfer (facial landmarks as reference) style to frame 1, propagate to others by optical flow ● Does not capture texture. and transfer after that. Convolutional Neural Networks 8

Convolutional Neural Networks 9

Gatys, et.al. approach Convolutional Neural Networks 10

Gatys, et.al. results Convolutional Neural Networks 11

Convolutional Neural Networks 12

Convolutional Neural Networks 13

The improved model ● Modified loss function – All levels of detail to maintain structure (reduce deformation) – Gain maps: local statistics → between input and example alignment!!! ● Alignment: – Find landmarks (66) → align: ● morphing (eyes / mouth) ● sift-flow (facial contour) E: example I: input M: modified w/ Gain Map O: output Convolutional Neural Networks 14

Γ = 1, 10, 100 Convolutional Neural Networks 15

Gain maps only Convolutional Neural Networks 16

Results ● Gain map clamp: [0.75, 5] ● Activation of the parameters for the 3th and 4 th convolutional layer only ● Style / identity: ( Γ = 100) ● Comparison against original Gatys, et.al. And a modified version 1) Preserves input identity 2) Transfer local spatial color distribution of the style 3) Reduce deformation during texture transfer Convolutional Neural Networks 17

Clamping + Stages Convolutional Neural Networks 18

Ghosting Convolutional Neural Networks 19

Alignment Convolutional Neural Networks 20

Convolutional Neural Networks 21

Convolutional Neural Networks 22

Convolutional Neural Networks 23

Convolutional Neural Networks 24

Video [video] Convolutional Neural Networks 25

Discussion ● CNN + spatial constrains: improved results ● Needs only one example and works with any style ● Experimented with a wide range of portraits and styles ● Temporal coherent results for video ● Very specific application with limitations ● CNN are powerful, but it could (should) be combined with robust classic strategies for better results ● Each application needs to adapt the “trendy” tech and use specific knowledge base Convolutional Neural Networks 26

The end Convolutional Neural Networks 27