Compressive Classification (Machine Learning without learning) - - PowerPoint PPT Presentation

Compressive Classification

(Machine Learning without learning) Vincent Schellekens Laurent Jacques

• 1

!2

Machine Learning

θ

Mathematical model

xN

=

n

features

N

examples

X

dataset Learning

!3

Machine Learning

Unsupervised

θ

Mathematical model

xN

=

n

features

N

examples

X

dataset Learning

• Clustering

“explain” the data

• Dimensionality reduction
• Autoencoder, GAN, SOM…

E.g.,

θ

θ

PCA

!4

Machine Learning

Unsupervised Supervised

θ

Mathematical model

xN

=

n

features

N

examples

X

dataset Learning

• Clustering

“explain” the data

• Dimensionality reduction
• Autoencoder, GAN, SOM…

E.g.,

θ

Mathematical model

xN

=

n

features

X

dataset Learning

• Classification

“generalize” the data

• Regression

E.g.,

θ

θ

target output

yN

θ

θ

PCA

!5

Compressive Learning

Usual machine learning

Learning

θ

X

Multiple passes over X

!6

Compressive Learning

Usual machine learning Usual machine learning

Learning

θ

… large memory & training time!

X

… … Large means…

N

Multiple passes over X

!7

Compressive Learning

Usual machine learning Compressive Learning

Learning

θ

… large memory & training time!

X

… … Large means…

N

“ S k e t c h i n g ”

θ

X

e.g., [Gribonval-CL]

Multiple passes over X

Single pass over X

“Learning”

zX

Dataset sketch (summary)

m

“observations”

m ' size(θ) ⌧ Nn

!8

Compressive Learning

Usual machine learning Compressive Learning

θ

… constant memory & training time!

X

… Large means…

N

e.g., [Gribonval-CL]

zX

Dataset sketch (summary)

m

“observations”

m ' size(θ) ⌧ Nn

Single pass over X

Usual machine learning

Learning

θ

… large memory & training time!

X

… … Large means…

N

Multiple passes over X

“ S k e t c h i n g ” “Learning”

!9

Previously on Compressive Learning…

Unsupervised Compressive Learning Unsupervised ML Supervised ML

• Compressive K-Means [Keriven-CKM]
• Compressive GMM estimation [Keriven-GMM]
• Compressive PCA [Gribonval-CL]

Supervised Compressive Learning ????

!10

In this talk…

Unsupervised Compressive Learning Unsupervised ML Supervised ML Supervised Compressive Learning

Compressive Classification

(a proof of concept)

• Compressive K-Means [Keriven-CKM]
• Compressive GMM estimation [Keriven-GMM]
• Compressive PCA [Gribonval-CL]

!11

Unsupervised Compressive Learning

In this talk…

Unsupervised ML Supervised ML Supervised Compressive Learning

Compressive Classification

(a proof of concept)

Preliminary 1:

(Unsupervised) Compressive Learning Basics

• Compressive K-Means [Keriven-CKM]
• Compressive GMM estimation [Keriven-GMM]
• Compressive PCA [Gribonval-CL]

!12

Compressing a dataset

Model:

θ

X

xi ∼ P

x

P(x)

??

What you get (empirical distribution):

x

xi

“simple” (sparse)

P

ˆ PX(x) := 1 N

N

X

i=1

δ(x − xi)

VS

!13

Compressing a dataset

Model:

θ

X

xi ∼ P

“simple” (sparse)

P

x

P(x)

??

What you get (empirical distribution):

x

xi

We don’t care about individual realisations Object of interest: the distribution !

P

distribution

ˆ PX(x) := 1 N

N

X

i=1

δ(x − xi)

VS

!14

Compressing a dataset

“Sketching”

θ

X

“Learning”

zX

Dataset sketch (summary)

m

distribution

x

ˆ PX(x)

“Signal” of interest

!15

Compressing a dataset

“Sketching”

θ

X

“Learning”

zX

Dataset sketch (summary)

m

distribution

x

ˆ PX(x)

Generalized moments Linear sensing Inverse problem Moment matching

A(P) := Ex∼Pf(x)

zX := A( ˆ PX) = 1 N X

xi∈X

f(xi) ∈ Cm

'

“Signal” of interest f computes m “features” of the data, to be averaged (“pooled”)

f

m

CS-inspired!

!16

Compressing a dataset

“Sketching”

θ

X

“Learning”

zX

Dataset sketch (summary)

m

distribution

x

ˆ PX(x)

Generalized moments Linear sensing Inverse problem Moment matching E.g. Random Fourier Features

A(P) := Ex∼Pf(x)

zX := A( ˆ PX) = 1 N X

xi∈X

f(xi) ∈ Cm

'

f(x)RFF = ⇥ exp(iωT

j x)

⇤m

j=1

“Signal” of interest f computes m “features” of the data, to be averaged (“pooled”)

f

m

CS-inspired!

Geometric interpretation

x

P(x)

κ(x, P)

x

hf(x), f(x0)i ' κ(x, x0)

approximates a kernel

Hκ

associated with a RKHS

f

Hκ

“Mean Map”

κ(·, P) := Ex0⇠Pκ(·, x0)

“Hκ = span ({κ(·, u)}) ”

[Smola]

Geometric interpretation

x

P(x)

κ(x, P)

x

hf(x), f(x0)i ' κ(x, x0)

approximates a kernel

Hκ

associated with a RKHS

f

Hκ

“Mean Map”

κ(·, P) := Ex0⇠Pκ(·, x0)

“Hκ = span ({κ(·, u)}) ”

f(x)RFF = ⇥ exp(iωT

j x)

⇤m

j=1

ωj ∼ Λ

hf(x)RFF, f(x0)RFFi

'

κ(x, x0) = K(u = x − x0) = (FΛ)(u)

with :

e.g.,

κ

Geometric interpretation

x

P(x)

κ(x, P)

x

hf(x), f(x0)i ' κ(x, x0)

approximates a kernel

Hκ

associated with a RKHS

f

Hκ

“Mean Map”

κ(·, P) := Ex0⇠Pκ(·, x0)

m → ∞

“Hκ = span ({κ(·, u)}) ”

Hκ

Key fact: the map approximatively preserves the geometry of

A : P → Ex∼Pf(x) ∈ Cm

Geometric interpretation

x

P(x)

κ(x, ˆ PX)

κ(x, P)

x

x

ˆ PX(x)

x

hf(x), f(x0)i ' κ(x, x0)

approximates a kernel

Hκ

associated with a RKHS

f

Hκ

“Mean Map”

κ

∗

'

κ(·, P) := Ex0⇠Pκ(·, x0)

N → ∞

m → ∞

“Hκ = span ({κ(·, u)}) ”

Hκ

Key fact: the map approximatively preserves the geometry of

A : P → Ex∼Pf(x) ∈ Cm

• Compressive K-Means [Keriven-CKM]
• Compressive GMM estimation [Keriven-GMM]
• Compressive PCA [Gribonval-CL]

!21

Unsupervised Compressive Learning

In this talk…

Unsupervised ML Supervised ML Supervised Compressive Learning

Compressive Classification

(a proof of concept)

Preliminary 2:

(Uncompressed) Classification Basics

!22

Classification problem

X

??

target

• utput

Classifier

ˆ y(x0)

x0

K

classes

!23

Classification problem

Model:

X

??

target

• utput

(xi, yi) ∼

K

X

k=1

pkPk(x)

Classifier

ˆ y(x0)

x0

K

classes

!24

Classification problem

Model:

X

??

target

• utput

(xi, yi) ∼

K

X

k=1

pkPk(x)

P(x|y = k)

Classifier

ˆ y(x0)

x0

P2

P1

Conditional

K

classes

!25

Classification problem

Model:

X

x

??

target

• utput

(xi, yi) ∼

K

X

k=1

pkPk(x)

P [y = k]

P(x|y = k)

Classifier

ˆ y(x0)

x0

P2

P1

p1

p2

Prior Conditional

P(x)

K

classes

!26

Optimal classifier

Model:

X

x

??

target

• utput

(xi, yi) ∼

K

X

k=1

pkPk(x)

P [y = k]

P(x|y = k)

Classifier

ˆ y(x0)

x0

P2

P1

p1

p2

Prior Conditional

P(x)

Best we can do (oracle knowledge):

MAP (or Bayes) classifier

ˆ yMAP := arg max

k

pkPk(x0)

K

classes

!27

Optimal classifier

Model:

X

x

??

target

• utput

(xi, yi) ∼

K

X

k=1

pkPk(x)

P [y = k]

P(x|y = k)

Classifier

ˆ y(x0)

x0

P2

P1

p1

p2

Prior Conditional

P(x)

Best we can do (oracle knowledge):

MAP (or Bayes) classifier

ˆ yMAP := arg max

k

pkPk(x0)

Hard to estimate in practice!

You get this:

K

classes

!28

In this talk… (finally)

Unsupervised Compressive Learning Unsupervised ML Supervised ML Supervised Compressive Learning

Compressive Classification

(a proof of concept)

• Compressive K-Means [Keriven-CKM]
• Compressive GMM estimation [Keriven-GMM]
• Compressive PCA [Gribonval-CL]

!29

Compressive Classifier

Observation (sketching) phase

One sketch per class! i.e., sketch the conditionals!

k ∈ {1, ..., K}

zX1

X1

X2

zX2

'

A(Pk) = Ex∼Pkf(x)

Km

coefficients to store

“Sketching”

zXk = 1 Nk X

xi∈Xk

f(xi)

Km ⌧ Nn

Efficient compression when

!30

Compressive Classifier

Observation (sketching) phase

One sketch per class! i.e., sketch the conditionals!

k ∈ {1, ..., K}

zX1

X1

X2

zX2

'

A(Pk) = Ex∼Pkf(x)

“Sketching”

ˆ yCC(x0) := arg max

k

ˆ pk · hf(x0), zXki

zXk = 1 Nk X

xi∈Xk

f(xi)

Classification phase

approximated prior sketch class correlation

x0

f(x0)

h·, ·i

zX1

zX2

max

f

ˆ yCC(x0)

Nk N

p1

p2

!31

Interpretation

x

Hκ

κ

∗

hf(x), f(x0)i ' κ(x, x0)

κ(·, P) := Ex0⇠Pκ(·, x0)

arg max

k

ˆ pk · κ(x0, ˆ PXk)

ˆ PX1

ˆ PX2

m → ∞

!32

Interpretation

ˆ yMAP = arg max

k

pkPk(x0)

x

Hκ

'

κ

∗

hf(x), f(x0)i ' κ(x, x0)

κ(·, P) := Ex0⇠Pκ(·, x0)

arg max

k

ˆ pk · κ(x0, ˆ PXk)

arg max

k

pk · κ(x0, Pk)

ˆ PX1

ˆ PX2

N → ∞

m → ∞

!33

Interpretation

ˆ yMAP = arg max

k

pkPk(x0)

x

Hκ

'

κ

∗

hf(x), f(x0)i ' κ(x, x0)

κ(·, P) := Ex0⇠Pκ(·, x0)

CC approximates the MAP classifier projected into the RKHS !

Hκ

arg max

k

ˆ pk · κ(x0, ˆ PXk)

arg max

k

pk · κ(x0, Pk)

ˆ yCC = arg max

k

ˆ pk · hf(x0), zXki

'

arg max

k

pk · κ(x0, Pk)

ˆ PX1

ˆ PX2

N → ∞

m → ∞

!34

Pros and cons

ˆ yCC = arg max

k

ˆ pk · hf(x0), zXki

'

arg max

k

pk · κ(x0, Pk)

Pro

• Cheap to “learn”: you only observe the

dataset (“compressive k-NN”)

• Cheap to evaluate: f (once), +, * and max
• Easy to parallelize/update: suited to massive

datasets, distributed (sensitive?) datasets, data streams, and data augmentation (almost free!)

• MAP in RKHS interpretation is seducing,

but it is an asymptotical result… zXk = 1 Nk X

xi∈Xk

f(xi)

then

f

!35

Pros and cons

ˆ yCC = arg max

k

ˆ pk · hf(x0), zXki

'

arg max

k

pk · κ(x0, Pk)

Con Pro

• Cheap to “learn”: you only observe the

dataset (“compressive k-NN”)

• Cheap to evaluate: f (once), +, * and max
• Easy to parallelize/update: suited to massive

datasets, distributed (sensitive?) datasets, data streams, and data augmentation (almost free!)

• MAP in RKHS interpretation is seducing,

but it is an asymptotical result…

• No guarantees (yet): generalization error

bounds, sample complexity bounds…

• Hyperparameters: choice of f (hence k) is

crucial!

• High-Dimensional data is expected to be

difficult zXk = 1 Nk X

xi∈Xk

f(xi)

then

f

f

κ

!36

Proof of concept (synthetical)

0.5 1 1.5 2 2.5 3 3.5 4

log10(m)

0.3 0.4 0.5 0.6 0.7 0.8 0.9

Test accuracy

var = 10e-3 var = 0.01 var = 0.1 var = 1 var = 10 Optimal (MAP)

1 1.5 2 2.5 3 3.5 4 4.5 5

log10(m)

0.3 0.4 0.5 0.6 0.7 0.8 0.9

Test accuracy

var = 10e-4 var = 10e-3 var = 0.1 var = 0.1 var = 1 Optimal (MAP)

• 4
• 2

2 4

• 4
• 2

2 4

• 2
• 1

1 2

• 2
• 1

1 2

Random Fourier Features sketch with Gaussian kernel

!37

Proof of concept (real)

Random Fourier Features sketch with Gaussian kernel Error rates (%) with SD

• n train and test sets

Standard learned classifier Compressive classifier

!38

In this talk…

Unsupervised Compressive Learning Unsupervised ML Supervised ML Supervised Compressive Learning

Compressive Classification

(a proof of concept) What about HD data such as images?

• Compressive K-Means [Keriven-CKM]
• Compressive GMM estimation [Keriven-GMM]
• Compressive PCA [Gribonval-CL]

!39

Sketching images

Up to now, the sketch function f used are Random Fourier Features For images, a kernel based on the pixel-wise difference does not make so much sense… Idea: capture the image structure with a (random) Convolutional Neural Network architecture

f

f(x)RFF = ⇥ exp(iωT

j x)

⇤m

j=1

κ(x, x0) = K(x − x0)

⇔

Random CNN have been shown to be surprisingly meaningful features for image processing tasks E.g., [Cho], [Giryes], [Rosenfeld], … Depends only on signal difference

f(x)CNN = CNNθ(x)

!40

Sketching images

Up to now, the sketch function f used are Random Fourier Features For images, a kernel based on the pixel-wise difference does not make much sense… Idea: capture the image structure with a (random) Convolutional Neural Network architecture

f

f(x)RFF = ⇥ exp(iωT

j x)

⇤m

j=1

κ(x, x0) = K(x − x0)

⇔

“Not that random” (Work In Progress)…

f(x)CNN = CNNθ(x)

!41

Some perspectives

• Explore the features/kernel choice?
• Learn the kernel from a small data sample (distilled sensing)?
• (Non-asymptotical) formal guarantees?
• Is this the best we can do with the sketch?
• Regression? Other supervised tasks?
• …

ˆ yCC = arg max

k

ˆ pk · hf(x0), zXki

'

arg max

k

pk · κ(x0, Pk)

zXk = 1 Nk X

xi∈Xk

f(xi)

then

!42

(Some) references

• [Keriven-CKM] N. Keriven, N. Tremblay,
• Y. Traonmilin, R. Gribonval, “Compressive

K-Means”, ICASSP 2017

• [Keriven-GMM] N. Keriven, A. Bourrier, R. Gribonval, P

. Perez, “Sketching for Large-Scale Learning of Mixture Models”, Information and Inference, 2017

• [Gribonval-CL] R. Gribonval, G. Blanchard, N. Keriven,
• Y. Traonmilin,

“Compressive Statistical Learning with Random Feature Moments”, ArxiV, 2017

• [Giryes] R. Giryes, G. Sapiro, A.M. Bronstein, “Deep Neural Networks with

Random Gaussian Weights: A Universal Classification Strategy?”, IEEE Transactions on Signal Processing, 2016

• [Cho]
• Y. Cho, L.K. Saul, “Kernel methods for deep learning”, NIPS 2009
• [Rosenfeld] A. Rosenfeld, J.K. Tsotsos, “Intriguing Properties of Randomly

Weighted Networks: Generalizing While Learning Next to Nothing”, ArxiV, 2018

• [Smola] A. Smola, A. Gretton, L. Song, B. Scholkopf, “A Hilbert space embedding

for distributions”, International Conference on Algorithmic Theory, 2007