Photo Editing With Generative Adversarial Networks (GANs) GTC, May - - PowerPoint PPT Presentation

β–Ά
photo editing with generative
SMART_READER_LITE
LIVE PREVIEW

Photo Editing With Generative Adversarial Networks (GANs) GTC, May - - PowerPoint PPT Presentation

Oct 05, 2022 337 likes β€’606 views

Photo Editing With Generative Adversarial Networks (GANs) GTC, May 2017. Greg Heinrich. PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects

slide-1
SLIDE 1

GTC, May 2017. Greg Heinrich.

Photo Editing With Generative Adversarial Networks (GANs)

slide-2
SLIDE 2

2 2

GAN: WHAT IS A GENERATIVE MODEL?

Learn from Shakespeare novels:

http://karpathy.github.io/2015/05/21/rnn- effectiveness/

Produce:

PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep.

In Machine Learning A generative model learns to generate samples that have the same characteristics as the samples in the dataset.

PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep.

slide-3
SLIDE 3

3 3

BASIC REMINDER: BACKPROP

Output of each neuron π‘˜ of layer π‘š :

β„Žπ‘˜

π‘š = πœ’ 𝑨 π‘˜ π‘š = πœ’ π‘₯π‘—π‘˜ π‘š β„Žπ‘— π‘šβˆ’1 + 𝑐 π‘˜ π‘š 𝑗

Gradient of E with respect to each weight:

πœ–πΉ πœ–π‘₯π‘—π‘˜

π‘š =

πœ–πΉ πœ–π‘¨π‘˜

π‘š

πœ–π‘¨π‘˜

π‘š

πœ–π‘₯π‘—π‘˜

π‘š =

πœ–πΉ πœ–π‘¨π‘˜

π‘š β„Žπ‘—

π‘šβˆ’1

Calculation of

πœ–πΉ πœ–π‘¨π‘˜

π‘š: πœ–πΉ πœ–π‘¨π‘˜

π‘š =

πœ–πΉ πœ–π‘¨π‘™

π‘š+1

𝑙 πœ–π‘¨π‘™

π‘š+1

πœ–π‘¨π‘˜

π‘š =

πœ–πΉ πœ–π‘¨π‘™

π‘š+1

πœ–π‘¨π‘™

π‘š+1

πœ–β„Žπ‘˜

π‘š

πœ–β„Žπ‘˜

π‘š

πœ–π‘¨π‘˜

π‘š

𝑙

= πœ–πΉ πœ–π‘¨π‘™

π‘š+1 π‘₯ π‘˜π‘™ π‘š+1πœ’β€² 𝑨 π‘˜ π‘š 𝑙

= πœ’β€² 𝑨

π‘˜ π‘š πœ–πΉ

πœ–π‘¨π‘™

π‘š+1 π‘₯ π‘˜π‘™ π‘š+1 𝑙

Calculating

πœ–πΉ πœ–π‘₯π‘—π‘˜

π‘š iteratively

Multivariate chain rule Chain rule

Chain rule

πœ–π‘¨π‘˜

π‘š

πœ–π‘₯π‘—π‘˜

π‘š only depends on β„Žπ‘— π‘šβˆ’1

Calculated during forward prop

slide-4
SLIDE 4

4 4

GAN: PLAYING THE ADVERSARIAL GAME

Learning on a corpus of images Let’s play a game opposing two agents:

  • The Generator, a little imp in the computer who paints images.
  • The Discriminator: you are collectively responsible for playing the

Discriminator. The game master (me) randomly picks images from either the corpus or the Generator and shows them to the Discriminator. The goal of the Discriminator is to identify the source of the images: real (from the corpus) or fake (painted by the little imp). The goal of the Generator is to fool the Discriminator.

slide-5
SLIDE 5

5 5

PLAYING THE ADVERSARIAL GAME

* veelhoek is the articulation

  • f a ubiquitous item in the

language of a tiny country in Europe that is well known for the inferior quality of its cheese.

Is this a veelhoek* from our corpus?

Yes, this red square is a veelhoek!

Note: you don’t have to know what a veelhoek is, you will learn through examples!

slide-6
SLIDE 6

6 6

No, those squiggly lines aren’t right!

PLAYING THE ADVERSARIAL GAME

Is this a veelhoek from our corpus?

slide-7
SLIDE 7

7 7

PLAYING THE ADVERSARIAL GAME

Is this a veelhoek from our corpus?

Yes, even though it’s blue and tiny!

slide-8
SLIDE 8

8 8

PLAYING THE ADVERSARIAL GAME

Is this a veelhoek from our corpus?

No, those rounded corners are a giveaway!

slide-9
SLIDE 9

9 9

No, but it’s a very good fake!

PLAYING THE ADVERSARIAL GAME

Is this a veelhoek from our corpus?

slide-10
SLIDE 10

10 10

No, it’s the same fake as before!

PLAYING THE ADVERSARIAL GAME

Is this a veelhoek from our corpus?

slide-11
SLIDE 11

11 11

PLAYING THE ADVERSARIAL GAME

Is this a veelhoek from our corpus?

No, but it’s a very creative fake!

slide-12
SLIDE 12

12 12

THE LATENT REPRESENTATION

From features to images

A veelhoek is characterized by three features:

  • colour,
  • size,
  • number of faces

This set of features is known as the β€œLATENT REPRESENTATION”. We can generate many real-looking veelhoeks by randomly picking reasonable values

  • f each feature:
slide-13
SLIDE 13

13 13

THE LATENT REPRESENTATION

Arithmetic in latent space

We can perform operations in latent space, have them reflected in feature space:

1 2 π‘šπ‘π‘ π‘•π‘“ 𝑠𝑓𝑒 3 𝑔𝑏𝑑𝑓𝑑 + π‘‘π‘›π‘π‘šπ‘š π‘•π‘ π‘“π‘“π‘œ 5 𝑔𝑏𝑑𝑓𝑑 = 𝑛𝑓𝑒𝑗𝑣𝑛 π‘§π‘“π‘šπ‘šπ‘π‘₯ 4 𝑔𝑏𝑑𝑓𝑑

Equivalently:

1 2 𝑠𝑓𝑒 π‘šπ‘π‘ π‘•π‘“ 3 𝑔𝑏𝑑𝑓𝑑 + π‘•π‘ π‘“π‘“π‘œ π‘‘π‘›π‘π‘šπ‘š 5 𝑔𝑏𝑑𝑓𝑑 = +

slide-14
SLIDE 14

14 14

THE GAN SET-UP

Connecting the Discriminator to the Generator and the Dataset

Random Latent vector

slide-15
SLIDE 15

15 15

GAN: NETWORK TOPOLOGY

Radford (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv:1511.06434

Generator Discriminator

slide-16
SLIDE 16

16 16

TRAINING A GAN ON CELEBRITY FACES*

* CelebFaces dataset

Generating new faces by picking random values of the latent vector

slide-17
SLIDE 17

17 17 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

ANALOGIES

man is to woman as king is to queen

Man Blond Hair Blue Eyes Smile Looking Left Pointy Nose Top Right

  • +

+ +

Bottom Left

+ + +

  • +

Subtract Top Left

+

  • Bottom Right

+

  • +

+

Reproduction of the famous β€œking + woman - man = queen” analogy on faces:

slide-18
SLIDE 18

18 18

MAPPING IMAGES TO LATENT VECTORS

Transfer learning: from Discriminator to Encoder

slide-19
SLIDE 19

19 19

IMAGE RECONSTRUCTIONS

Visualizing 𝐻 𝐹 𝑗𝑛𝑏𝑕𝑓

slide-20
SLIDE 20

20 20

ATTRIBUTES

The encoder 𝐹 may be used to calculate the latent vector for each attribute. For each 𝑏𝑒𝑒𝑠 in 𝑏𝑒𝑒𝑠𝑗𝑐𝑣𝑒𝑓𝑑 : 𝐽𝑏𝑒𝑒𝑠

+

= 𝑗𝑛 𝑏𝑒𝑒𝑠 and 𝐽𝑏𝑒𝑒𝑠

βˆ’

= 𝑗𝑛 𝑏𝑒𝑒𝑠 are sets of images w/wo the attribute 𝑨 𝑏𝑒𝑒𝑠 =

1 𝐽𝑏𝑒𝑒𝑠

+

𝐹(𝑗𝑛)

𝑗𝑛 ∈ 𝐽𝑏𝑒𝑒𝑠

+

βˆ’

1 𝐽𝑏𝑒𝑒𝑠

βˆ’

𝐹(𝑗𝑛)

𝑗𝑛 ∈ 𝐽𝑏𝑒𝑒𝑠

βˆ’

It is then straightforward to add or remove attributes from an image:

Calculating attribute vectors

From left to right: original image (OI); OI + β€œyoung” attribute; OI - β€œblond hair” + β€œblack hair”; OI - β€œsmile”; OI + β€œmale” + β€œbald”.

slide-21
SLIDE 21

21 21

PLAYING WITH ATTRIBUTES

slide-22
SLIDE 22

22 22

EXTRACTING ATTRIBUTES

…from portraits of illustrious people

slide-23
SLIDE 23

23 23

DEGENERATOR

Getting the essence of your dataset

After convergence, stop updating the discriminator:

slide-24
SLIDE 24

24 24

DATASET VISUALIZATION

Projecting latent vectors on a sphere

slide-25
SLIDE 25

25 25

THANK YOU

Questions?