CSC2412: Exponential Mechanism & Private PAC Learning Sasho - - PowerPoint PPT Presentation

SLIDE 1

CSC2412: Exponential Mechanism & Private PAC Learning

Sasho Nikolov

1

SLIDE 2

Classification Basics

SLIDE 3

The learning problem

Problem: develop an algorithm that classifies avocados into ripe and unripe. We have a big data set of avocado data. For each avocado, we have:

• colour, firmness, size, shape, skin texture, . . .
• ripe or not

From this data, we want to classify unseen avocados.

2 + I

• I

.

features label

Learn

a

rule

to

predict

label

from

features

SLIDE 4

The learning problem, formally

Model:

• Known data universe X and an unknown probability distribution D on X
• Known concept class C and an unknown concept c 2 C
• We get a dataset X = {(x1, c(x1), . . . , (xn, c(xn)}}, where each xi is an

independent sample from D. Goal: Learn c from X.

3

all possible setting

to the features

• id
• ⇐

All allow

#rules

to my

• A"YYel:#Facility)

:

features

to label

.

Iiia

't done

"plIE.fi

features

some

cec

can

QQ.e.ge

,

produce

all effete

features -

E

• g

.

an

avocado

is

ripe

approximation of

e

/ iff

colour

= Attu

ye

and

firmness

= medium

SLIDE 5

The goal, formally

The error of a concept c0 2 C is LD,c = Px⇠D(c0(x) 6= c(x)). We want an algorithm M that outputs some c0 2 C and satisfies P(LD,c(M(X))  α) 1 β.

4

µ

Fraction

• f

the

population

labeled

↳HWA

incorrectly by

c

'

Floss of

c

' ( w . r. t .

D

,

c )

• n input

X

• tix.

. . CHD , . . .,Kn , CCN)

sampled

iid

from

D

^

→ output of UK)

misclassifies

Ed fraction

• f the population

% taken

• ver

randomness

in choosing

X and any

randomness

• f µ

Probably Approximately

Correct

learning

( PAC)

[ Valiant]

SLIDE 6

Empirical risk minimization

Issue: We want to find arg minc02C LD,c(c0), but we do not know D, c. Solution: Instead we solve arg minc02C LX(c0), where LX(c0) = |{i : c0(xi) 6= c(xi)}| n is the empirical error. Theorem (Uniform convergence) Suppose that n ln(|C|/β)

2α2

. Then, with probability 1 β, max

c02C LX(c0) LD,c(c0)  α. 5

⇒

pnpnoxiuate minimizerok LD ,e Population ↳c4=O cuukuol

:L ,

I

fraction of

pts

in X

Lx empirical

loss misclassified by

c

'

ckuowil

p

→ Pop and emp

. loss

are

close

for

Kc

'

c- C

uoeetoung

'EE1Yt

WHY

'hHfhH'

tic:{ co ,

• c. Lxcilth

Other

versions

for infinite

C

, e.g ,

VC

• dimension

SLIDE 7

Private learning

In private PAC learning, we require that

• when X is a sample of iid labeled data points, we learn the correct concept, as in

stadard PAC learning;

• the learning algorithm is ε-differentially private for any labeled data set

X 2 (X ⇥ C)n.

6

• unzipproximately

T arin put X outputs

ele C

Privacy

must

• hold

even

if data X

f

• X. x

'

neighbouring

t

s

'

• C

not

iid

PINCH

e- S )

t

ee

. PINK

' ) c- S )

SLIDE 8

Want to

do

ERM

we

e

• DP
• i. e.

( approximately )

minimize

↳

( c ' )

=

I

i

:

Citi )

#

c' exit } I

#

• ver

c

'

C- C

'How

can

we

use

Laplace

noise

mechanism

for

this

?

Exercise

: analyze

↳ co )

is

a counting query

this f

we

could

release

answers

to

all

counting queries

C- 94

,

. . , eh }

{ 1×14

. )

, 4h27 ,

. - .

. ↳ 141 }

SLIDE 9

Exponential mechanism

SLIDE 10

Private ERM

We want to solve arg minc02C LX(c0). How do we minimize with differential privacy? Sample concepts with less error with higher probability P(M(X) = c0) / exp ⇣ εn 2 LX(c0) ⌘

7

• r

D

ptoportionat to

SLIDE 11

Exponential Mechanism

General set-up: score function u : X ⇥ Y ! R Sensitivity ∆u = max

y2Y max X⇠X 0 |u(X, y) u(X 0, y)|.

The mechanism Mexp(X) which outputs a random Y so that P(Y = y) = eεu(X,y)/2∆u P

z2Y eεu(X,z)/2∆u

is ε-differentially private

8

• utput space

u (X

, y)

=

" how good of

Ban

an

• utput

is y

Goal

: given

X

, find

argq.mg

UH

, y )

for

the dataset

X ?

"

How

different

can

the

score

be between neighbouring

X

, X

'

→ normalizing

factor

SLIDE 12

Privacy analysis

9

Enough

to show

: H X - t

'

PINCH

• y )
• tye 's

p¥=y

. )

'

e '

in::÷÷

, .e*eu"¥:"#t¥÷::÷÷:

E eEK.EE/Z--eE

SLIDE 13

Accuracy of the exponential mechanism

OPT(X) = max

y2Y u(X, y)

Then, for the output Y = Mexp(X), P(u(X, Y )  OPT(X) t) 

10

ZL

• utput

y*

achieves

OPTLXIEECOPTCX

)

• t )/2gy
• Eeg eeuk.at/2aaoHy:UCt,y

)

a- OPT

• tf

z

ee 'd

thou

s

eel

OPT

• t ) KDU

. (left

• l )

e¥sq-

⇐ e- ethos ?

s

e

• Ethan

.

peg )

191

SLIDE 14

Private Learning

SLIDE 15

Unknown

distribution

D

• u

known R

Unknown

c

in

a

known

concept

class

C

Data

set

• X
• 41K

, chill

,

. - , Hm

, um }

where

is

,

. . .

, Xu

• ii. D

LD

, etc

' )

• app

( cent

c' Ch ) / ↳ ( c

. ) . . Ki:cHniHdHi#

If

n

z

lull clip ) I

←

then

we prob z

t - p,

Kcie

C

! Lo

, ,

Cc

' )

e- ↳ kilt d

• Erath mechanism

:

sample

ye 9

we prob

.

proportional

to expleulkylb.su)

Au

• jug flag, lucky)
• ult

'

. g) I

SLIDE 16

Putting things together

A concept class C can be learned by an eps-differentially private mechanism when the sample size is n max ⇢4 ln(2|C|/β) εα , 2 ln(2|C|/β) α2

• 11

EEE

Use

exp

.

mechanism

with

B

• C

with ult.ci/---LxT-

With

. prob

?

I -

R2

,

c

'

• utput

by

heap (x)

has

↳ ( C

' ) Etz . By

unit . convergence , we prob z

t - Bf , LD

, etc

' )

• Lyla ) EE

SLIDE 17

Putting things together

12

MIX

, c.)

=

• ↳ ( c ' )

;

Du

= th ;

sample

c' e

C at prof

prop

.

to etpl

• en Lik 't)

Optfx)

• Max
• Lik 't

stop

c' c- C

=

• min

Lx ( c ')

• O

C ' c- C

IPI ↳ ( MH )) z E)

= plulx , UK)) EOPTKI

• E )

E

e

• Edm

. 14 spy

if

u z 4lntp)

Ed

+ by YwYfg

• III; )

w/ prob

I

• TE

.

↳

. . ( UCH) thrill CH) -172

upset

? ' -B drink =L

.