Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 - - PowerPoint PPT Presentation

▶
convolutional neural networks part iii
SMART_READER_LITE
LIVE PREVIEW

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 - - PowerPoint PPT Presentation

May 13, 2023 166 likes •446 views

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional

slide-1
SLIDE 1

Convolutional Neural Networks 1

Convolutional Neural Networks (Part III)

08, 10 & 17 Nov, 2016

  • J. Ezequiel Soto S.

Image Processing 2016

  • Prof. Luiz Velho
slide-2
SLIDE 2

Convolutional Neural Networks 2

Summary & References

08/11 ImageNet Classification with Deep Convolutional Neural Networks

2012, Krizhevsky et.al. [source]

10/11 Going Deeper with Convolutions

2015, Szegedy et.al. [source]

17/11 Painting Style Transfer for Head Portraits using Convolutional Neural Networks

2016, Selim & Elgharib [source]

+ A Neural Algorithm of Artistic Style

2015, Gatys et.al. [source]

+ Image Style Transfer Using Convolutional Neural Networks

2016, Gatys et.al. [source]

slide-3
SLIDE 3

Convolutional Neural Networks 3

Painting Style Transfer for Head Portraits using Convolutional Neural Networks

Selim & Elgharib, 2016

slide-4
SLIDE 4

Convolutional Neural Networks 4

Outline

  • Motivation
  • Related work

– Painting Transfer – Color Transfer – Extension to Video

  • Painting trough CNNs
  • Example Driven Spatial Constrains

– Modified feature maps – Solving for the painting

  • Results

– Still images – Extension to video

  • Discussion
slide-5
SLIDE 5

Convolutional Neural Networks 5

Motivation

  • Usual style-transfer techniques often deform facial structures: no spatial

constrains.

→ Human perception is very sensitive to changes in faces.

  • Other strategies:

– facial geometry (limited artistic styles: graphite & sketch painting) – image analogies (requires paired images: limited applicability) – color transfer: power maps (structure and color / no texture)

photo example Gatys, et.al. Selim, et.al.

slide-6
SLIDE 6

Convolutional Neural Networks 6

Related: Painting Transfer

  • Stroke based

– Simulates brush-stroke

placement process

– Brush attributes: scale,

  • rientation, color, opacity
  • Texture transfer

– Follow sample textures – Texture synthesis

  • Parts transfer

– Parse the input image in parts – Reconstruction from parts

database

Example: Gatys et.al. → Facial deformation!!!

– Facial landmarks as

guidance (AAM / ASM): eyes, eyebrows, nose, mouth, face outline

– Parts based techniques:

non parametric sampling, MRF and spatial smoothness in reconstruction: graphite & sketch

slide-7
SLIDE 7

Convolutional Neural Networks 7

slide-8
SLIDE 8

Convolutional Neural Networks 8

Related

Related: Color Transfer

  • Transfer color palette from

example to input

  • Not based in image

analogies

  • Power maps: local statistics.

Local distribution of light →

  • Spatial constraints: warping

(facial landmarks as reference)

  • Does not capture texture.

Related: Video

  • Applying painting frame by

frame generates: flickering & shower-door.

  • Flickering: temporal

inconsistencies.

  • Shower-door: texture drifts

away from the object.

  • Search better style, transfer

style to frame 1, propagate to others by optical flow and transfer after that.

slide-9
SLIDE 9

Convolutional Neural Networks 9

slide-10
SLIDE 10

Convolutional Neural Networks 10

Gatys, et.al. approach

slide-11
SLIDE 11

Convolutional Neural Networks 11

Gatys, et.al. results

slide-12
SLIDE 12

Convolutional Neural Networks 12

slide-13
SLIDE 13

Convolutional Neural Networks 13

slide-14
SLIDE 14

Convolutional Neural Networks 14

The improved model

  • Modified loss function

– All levels of detail to maintain

structure (reduce deformation)

– Gain maps: local statistics

between input and example → alignment!!!

  • Alignment:

– Find landmarks (66)

align: →

  • morphing (eyes / mouth)
  • sift-flow (facial contour)

E: example I: input M: modified w/ Gain Map O: output

slide-15
SLIDE 15

Convolutional Neural Networks 15

Γ = 1, 10, 100

slide-16
SLIDE 16

Convolutional Neural Networks 16

Gain maps

  • nly
slide-17
SLIDE 17

Convolutional Neural Networks 17

Results

  • Gain map clamp: [0.75, 5]
  • Activation of the parameters for the 3th and 4th convolutional

layer only

  • Style / identity: (Γ = 100)
  • Comparison against original Gatys, et.al. And a modified

version 1) Preserves input identity 2) Transfer local spatial color distribution of the style 3) Reduce deformation during texture transfer

slide-18
SLIDE 18

Convolutional Neural Networks 18

Clamping + Stages

slide-19
SLIDE 19

Convolutional Neural Networks 19

Ghosting

slide-20
SLIDE 20

Convolutional Neural Networks 20

Alignment

slide-21
SLIDE 21

Convolutional Neural Networks 21

slide-22
SLIDE 22

Convolutional Neural Networks 22

slide-23
SLIDE 23

Convolutional Neural Networks 23

slide-24
SLIDE 24

Convolutional Neural Networks 24

slide-25
SLIDE 25

Convolutional Neural Networks 25

[video]

Video

slide-26
SLIDE 26

Convolutional Neural Networks 26

Discussion

  • CNN + spatial constrains: improved results
  • Needs only one example and works with any style
  • Experimented with a wide range of portraits and styles
  • Temporal coherent results for video
  • Very specific application with limitations
  • CNN are powerful, but it could (should) be combined

with robust classic strategies for better results

  • Each application needs to adapt the “trendy” tech and

use specific knowledge base

slide-27
SLIDE 27

Convolutional Neural Networks 27

The end