Invariant-equivariant representation learning for multi-class data - - PowerPoint PPT Presentation

Invariant-equivariant representation learning for multi-class data

Ilya Feige Faculty

➔

High-level introduction

Invariant-equivariant representation learning

2

What? Want to represent the class and the data instance separately This work is about disentangling representations. We present a novel approach to an old problem.

Separating content from style

3

What? Want to represent the class and the data instance separately This work is about disentangling representations. We present a novel approach to an old problem.

Separating content from style

4

class, r

What? Want to represent the class and the data instance separately This work is about disentangling representations. We present a novel approach to an old problem.

Separating content from style

5

datapoint, (r, v)

class, r

What? Want to represent the class and the data instance separately Why?

• Classification
• Interpretability
• Object detection
• Topic modelling
• Style transfer
• Face swap
• …

This work is about disentangling representations. We present a novel approach to an old problem.

Separating content from style

6

datapoint, (r, v)

class, r

What? Want to represent the class and the data instance separately Why?

• Classification
• Interpretability
• Object detection
• Topic modelling
• Style transfer
• Face swap
• …

What else? This is not a new topic…

• Tenenbaum & Freeman. (2000)
• Reed et al. (2014)
• Cheung et al. (2014)
• Zhu et al. (2014)
• Radford et al. (2016)
• Chen et al. (2016)
• Makhzani et al. (2016)
• Siddharth et al. (2017)
• …

This work is about disentangling representations. We present a novel approach to an old problem.

Separating content from style

7

datapoint, (r, v)

class, r

The main idea

Inferring the class latent using strategic data routing Invariant (class) latent is deterministically calculated from “complementary” same-class examples:

8

{x1, x2, . . . , xm} − → ry

The main idea

Inferring the class latent using strategic data routing Invariant (class) latent is deterministically calculated from “complementary” same-class examples: Equivariant (instance) latent is stochastically inferred from both class and datapoint information:

9

{x1, x2, . . . , xm} − → ry

{ry, x} − → v

The main idea

Inferring the class latent using strategic data routing Invariant (class) latent is deterministically calculated from “complementary” same-class examples: Equivariant (instance) latent is stochastically inferred from both class and datapoint information:

10

{x1, x2, . . . , xm} − → ry

{ry, x} − → v

Inspired by GQNs (Eslami et al., 2018)

Some detail

Invariant-equivariant representation learning

11

Flash through the math

12

p

• {xn, yn}N

n=1

• =

N

Y

n=1

Z dvn drn pθ(xn|rn, vn) δ

• rn − ryn
• p(vn) p(yn)

Generative model

ryn = 1 m

m

X

i=1

fθinv(xi)

Inference model

qφ(v|ry, x) = N

• µφ(ry, x), σ2

φ(ry, x)I

• <latexit sha1_base64="+dvATW3nqowsdkwjBpECq3PAOUw=">ACPXicbVBLS8NAGNz4rPEV9ehlsRQqSEmqoBdB9KIXqWit0NSw2W7bpbtJ3N2IJeaHefJHePLkSVAQr17dtBF8DSwM/Px7Td+xKhUtv1ojI1PTE5NF2bM2bn5hUVraflchrHApI5DFoLH0nCaEDqipGLiJBEPcZafj9g8xvXBMhaRicqUFEWhx1A9qhGCktedbplZe4UY+m5etb4Q02btbhLnQ5Uj2MWHKcuj7tl0ef6VGmQ3oStrlKFcvq7l+lMXPatoV+wh4F/i5KQIctQ868FthzjmJFCYISmbjh2pVoKEopiR1Cy5sSQRwn3UJU1NA8SJbCXD61NY0kobdkKhX6DgUDW/TSISzngvk5md8nfXib+5zVj1dlpJTSIYkUCPFrUiRlUIcyqhG0qCFZsoAnCgurPQtxDAmGlCzd1C87vm/+S82rF2axUT7aKe/t5HwWwCtZAGThgG+yBQ1ADdYDBHXgCL+DVuDejTfjfRQdM/KZFfADxscn1lOuSg=</latexit>

Llab = Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤ + log p(y)

Lunlab = Eq(y|x) h Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤i − DKL ⇥ q(y|x)

• p(y)

⇤

Objective

Flash through the math

13

p

• {xn, yn}N

n=1

• =

N

Y

n=1

Z dvn drn pθ(xn|rn, vn) δ

• rn − ryn
• p(vn) p(yn)

Generative model

ryn = 1 m

m

X

i=1

fθinv(xi)

Inference model

qφ(v|ry, x) = N

• µφ(ry, x), σ2

φ(ry, x)I

• <latexit sha1_base64="+dvATW3nqowsdkwjBpECq3PAOUw=">ACPXicbVBLS8NAGNz4rPEV9ehlsRQqSEmqoBdB9KIXqWit0NSw2W7bpbtJ3N2IJeaHefJHePLkSVAQr17dtBF8DSwM/Px7Td+xKhUtv1ojI1PTE5NF2bM2bn5hUVraflchrHApI5DFoLH0nCaEDqipGLiJBEPcZafj9g8xvXBMhaRicqUFEWhx1A9qhGCktedbplZe4UY+m5etb4Q02btbhLnQ5Uj2MWHKcuj7tl0ef6VGmQ3oStrlKFcvq7l+lMXPatoV+wh4F/i5KQIctQ868FthzjmJFCYISmbjh2pVoKEopiR1Cy5sSQRwn3UJU1NA8SJbCXD61NY0kobdkKhX6DgUDW/TSISzngvk5md8nfXib+5zVj1dlpJTSIYkUCPFrUiRlUIcyqhG0qCFZsoAnCgurPQtxDAmGlCzd1C87vm/+S82rF2axUT7aKe/t5HwWwCtZAGThgG+yBQ1ADdYDBHXgCL+DVuDejTfjfRQdM/KZFfADxscn1lOuSg=</latexit>

Llab = Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤ + log p(y)

Lunlab = Eq(y|x) h Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤i − DKL ⇥ q(y|x)

• p(y)

⇤

Objective Generative model with 2 latent variables

Flash through the math

14

p

• {xn, yn}N

n=1

• =

N

Y

n=1

Z dvn drn pθ(xn|rn, vn) δ

• rn − ryn
• p(vn) p(yn)

Generative model

ryn = 1 m

m

X

i=1

fθinv(xi)

Inference model

qφ(v|ry, x) = N

• µφ(ry, x), σ2

φ(ry, x)I

• <latexit sha1_base64="+dvATW3nqowsdkwjBpECq3PAOUw=">ACPXicbVBLS8NAGNz4rPEV9ehlsRQqSEmqoBdB9KIXqWit0NSw2W7bpbtJ3N2IJeaHefJHePLkSVAQr17dtBF8DSwM/Px7Td+xKhUtv1ojI1PTE5NF2bM2bn5hUVraflchrHApI5DFoLH0nCaEDqipGLiJBEPcZafj9g8xvXBMhaRicqUFEWhx1A9qhGCktedbplZe4UY+m5etb4Q02btbhLnQ5Uj2MWHKcuj7tl0ef6VGmQ3oStrlKFcvq7l+lMXPatoV+wh4F/i5KQIctQ868FthzjmJFCYISmbjh2pVoKEopiR1Cy5sSQRwn3UJU1NA8SJbCXD61NY0kobdkKhX6DgUDW/TSISzngvk5md8nfXib+5zVj1dlpJTSIYkUCPFrUiRlUIcyqhG0qCFZsoAnCgurPQtxDAmGlCzd1C87vm/+S82rF2axUT7aKe/t5HwWwCtZAGThgG+yBQ1ADdYDBHXgCL+DVuDejTfjfRQdM/KZFfADxscn1lOuSg=</latexit>

Llab = Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤ + log p(y)

Lunlab = Eq(y|x) h Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤i − DKL ⇥ q(y|x)

• p(y)

⇤

Objective Generative model with 2 latent variables Deterministic, from same- class complementary data

Flash through the math

15

p

• {xn, yn}N

n=1

• =

N

Y

n=1

Z dvn drn pθ(xn|rn, vn) δ

• rn − ryn
• p(vn) p(yn)

Generative model

ryn = 1 m

m

X

i=1

fθinv(xi)

Inference model

qφ(v|ry, x) = N

• µφ(ry, x), σ2

φ(ry, x)I

• <latexit sha1_base64="+dvATW3nqowsdkwjBpECq3PAOUw=">ACPXicbVBLS8NAGNz4rPEV9ehlsRQqSEmqoBdB9KIXqWit0NSw2W7bpbtJ3N2IJeaHefJHePLkSVAQr17dtBF8DSwM/Px7Td+xKhUtv1ojI1PTE5NF2bM2bn5hUVraflchrHApI5DFoLH0nCaEDqipGLiJBEPcZafj9g8xvXBMhaRicqUFEWhx1A9qhGCktedbplZe4UY+m5etb4Q02btbhLnQ5Uj2MWHKcuj7tl0ef6VGmQ3oStrlKFcvq7l+lMXPatoV+wh4F/i5KQIctQ868FthzjmJFCYISmbjh2pVoKEopiR1Cy5sSQRwn3UJU1NA8SJbCXD61NY0kobdkKhX6DgUDW/TSISzngvk5md8nfXib+5zVj1dlpJTSIYkUCPFrUiRlUIcyqhG0qCFZsoAnCgurPQtxDAmGlCzd1C87vm/+S82rF2axUT7aKe/t5HwWwCtZAGThgG+yBQ1ADdYDBHXgCL+DVuDejTfjfRQdM/KZFfADxscn1lOuSg=</latexit>

Llab = Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤ + log p(y)

Lunlab = Eq(y|x) h Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤i − DKL ⇥ q(y|x)

• p(y)

⇤

Objective Generative model with 2 latent variables Deterministic, from same- class complementary data Standard VAE

Flash through the math

16

p

• {xn, yn}N

n=1

• =

N

Y

n=1

Z dvn drn pθ(xn|rn, vn) δ

• rn − ryn
• p(vn) p(yn)

Generative model

ryn = 1 m

m

X

i=1

fθinv(xi)

Inference model

qφ(v|ry, x) = N

• µφ(ry, x), σ2

φ(ry, x)I

• <latexit sha1_base64="+dvATW3nqowsdkwjBpECq3PAOUw=">ACPXicbVBLS8NAGNz4rPEV9ehlsRQqSEmqoBdB9KIXqWit0NSw2W7bpbtJ3N2IJeaHefJHePLkSVAQr17dtBF8DSwM/Px7Td+xKhUtv1ojI1PTE5NF2bM2bn5hUVraflchrHApI5DFoLH0nCaEDqipGLiJBEPcZafj9g8xvXBMhaRicqUFEWhx1A9qhGCktedbplZe4UY+m5etb4Q02btbhLnQ5Uj2MWHKcuj7tl0ef6VGmQ3oStrlKFcvq7l+lMXPatoV+wh4F/i5KQIctQ868FthzjmJFCYISmbjh2pVoKEopiR1Cy5sSQRwn3UJU1NA8SJbCXD61NY0kobdkKhX6DgUDW/TSISzngvk5md8nfXib+5zVj1dlpJTSIYkUCPFrUiRlUIcyqhG0qCFZsoAnCgurPQtxDAmGlCzd1C87vm/+S82rF2axUT7aKe/t5HwWwCtZAGThgG+yBQ1ADdYDBHXgCL+DVuDejTfjfRQdM/KZFfADxscn1lOuSg=</latexit>

Llab = Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤ + log p(y)

Lunlab = Eq(y|x) h Eq(v|ry,x) log p(x|ry, v) − DKL ⇥ q(v|ry, x)

• p(v)

⇤i − DKL ⇥ q(y|x)

• p(y)

⇤

Objective Generative model with 2 latent variables Deterministic, from same- class complementary data Standard VAE Standard ELBOs

Inference in pictures

17

Results

Invariant-equivariant representation learning

18

Inferred latent space on MNIST

19

Latent spaces disentangle The invariant latent learns to separate the classes The equivariant latent learns to ignore class information

Generations from various latent configurations (MNIST)

20

Samples from each class Equivariant interpolations, for multiple invariant latents Invariant steps, for multiple equivariant latents

Generations from various latent configurations (SVHN)

21

Samples from each class Equivariant interpolations, for multiple invariant latents Invariant steps, for multiple equivariant latents

Classification

Semi supervised (Using label-inference distribution)

22

Fully supervised (Using 0-parameter distance to nearest )

ry

Benchmark is always the equivalent network, +2 dropout layers, trained to classify the labelled data

Outlook

Invariant-equivariant representation learning

23

Outlook

Advantages

• f this approach
• A. Easy to implement
• B. Very little tuning needed

C.Reasonably intuitive D.Performs similarly to comparable approaches At Faculty, we are implementing this technique into our generic

• bject detection pipeline to extract invariant object representations

24

Disadvantages

• A. Requires some labels
• B. Does not achieve state-of-the-art

info@faculty.ai +44 20 3637 9415 54 Welbeck St, Marylebone, 
 London W1G 9XS, UK
 
 Follow us:

If you’re interested in finding out more about this work, or Faculty in general, please get in touch!

Thank you

25

ilya@faculty.ai Director of AI