PINFER: PRIVACY-PRESERVING INFERENCE DPM 2019 Luxembourg Sep. 26, - - PowerPoint PPT Presentation

▶
pinfer privacy preserving inference
SMART_READER_LITE
LIVE PREVIEW

PINFER: PRIVACY-PRESERVING INFERENCE DPM 2019 Luxembourg Sep. 26, - - PowerPoint PPT Presentation

Mar 05, 2024 416 likes •589 views

Innovation Centre PINFER: PRIVACY-PRESERVING INFERENCE DPM 2019 Luxembourg Sep. 26, 2019 Marc Joye Fabien Petitcolas MACHINE LEARNING AS A SERVICE GENERIC MODEL result Client Cloud (Server) 1 exchange of messages c 2019 OneSpan

slide-1
SLIDE 1

Innovation Centre

PINFER: PRIVACY-PRESERVING INFERENCE

DPM 2019 Luxembourg Sep. 26, 2019

Marc Joye Fabien Petitcolas

slide-2
SLIDE 2

MACHINE LEARNING AS A SERVICE — GENERIC MODEL

Client Cloud (Server)

result

1 exchange of messages

Innovation Centre c 2019 OneSpan Innovation Centre 2

slide-3
SLIDE 3

REQUIREMENTS AND SOLUTIONS Security requirements

  • The server learns nothing about the

client’s input

  • The server does not learn the output of

the calculation

  • The client learns nothing about the ML

model

Proposed solutions

Private evaluation for:

1 Linear regression 2 Logistic regression 3 Binary classification

  • Support Vector Machines (SVM)
  • requires a private comparison

protocol (e.g., DGK+)

4 Neural networks

  • Sign or ReLU activation functions
  • 1 interaction per layer

Innovation Centre c 2019 OneSpan Innovation Centre 3

slide-4
SLIDE 4

LINEAR PREDICTION MODEL

  • Input

1 Server’s ML model: θ = (θ0, . . . , θd) ∈ Rd+1 2 User’s feature vector: x = (1, x1. . . . , xd) ∈ {1} × Rd

  • Output

hθ(x) = g(θ⊺x) in many cases

Innovation Centre c 2019 OneSpan Innovation Centre 4

slide-5
SLIDE 5

LINEAR PREDICTION MODEL — EVALUATION FUNCTION g

Linear Regression [real-valued output] g = Id Logistic Regression [probability] g = σ where σ(s) =

exp(s) 1+exp(s)

Linear Classification [binary decision] g = sign Rectified linear unit (ReLU) [neural networks] g(s) =

  • if s < 0

s

  • therwise

Innovation Centre c 2019 OneSpan Innovation Centre 5

slide-6
SLIDE 6

LINEAR PREDICTION MODEL WITH ENCRYPTION

Model evaluation: ˆ y = g(θ⊺x)

Client (x) (pk, sk) Server (θ)

❶ Compute ⟦x⟧

⟦x1⟧, . . . , ⟦xd⟧, pk

❷ Compute ⟦g(θ⊺x)⟧

⟦g(θ⊺x)⟧

❸ Decrypt ⟦g(θ⊺x)⟧ Set ˆ y = g(θ⊺x)

Innovation Centre c 2019 OneSpan Innovation Centre 6

slide-7
SLIDE 7

LINEARLY HOMOMORPHIC ENCRYPTION

  • We only require linearly homomorphic encryption:

Encpk(m1) ⊞ Encpk(m2) = Encpk(m1 + m2)

  • NOT fully homomorphic encryption:

Encpk(m1) ⊞ Encpk(m2) = Encpk(m1 + m2) Encpk(m1) ⊡ Encpk(m2) = Encpk(m1 · m2)

  • Benefits
  • Simpler implementation
  • Faster computation

Innovation Centre c 2019 OneSpan Innovation Centre 7

slide-8
SLIDE 8

PRIVATE INNER PRODUCT

  • Since ⟦·⟧ is homomorphic

⟦θ⊺x⟧ = ⟦θ0 + d

i=1θixi

⟧ = ⟦θ0⟧ ⊞ ⟦θ1x1⟧ ⊞ · · · ⊞ ⟦θdxd⟧ and, for 1 ≤ i ≤ d, ⟦θixi⟧ = ⟦xi⟧ ⊞ · · · ⊞ ⟦xi⟧

  • θi times

:= θi ⊙ ⟦xi⟧

Example (Paillier’s cryptosystem)

  • ⟦m⟧ = (1 + N)m rN mod N2
  • ⟦m1 + m2⟧ = ⟦m1⟧ · ⟦m2⟧ mod N2
  • ⟦m1 − m2⟧ = ⟦m1⟧/⟦m2⟧ mod N2
  • a ⊙ ⟦m⟧ = ⟦m⟧a mod N2 =

⇒ ⟦θ⊺x⟧ requires d exponentiations modulo N2

Innovation Centre c 2019 OneSpan Innovation Centre 8

slide-9
SLIDE 9

IF EVALUATION FUNCTION g IS NON-LINEAR

  • g is non-linear but injective (e.g., σ)
  • Server computes ⟦θ⊺x⟧
  • Client obtains θ⊺x and simply applies g and learns no more

(by definition: g(a) = g(b) = ⇒ a = b)

  • g is non-linear and non-injective (e.g., sign, ReLU)
  • Use set of tools and tricks
  • DGK+ comparison protocol
  • Simple masking with a random value
  • Masking and scaling of inner product
  • Variant of oblivious transfer (two possible ciphers sent)
  • Dual setup
  • Server publishes pkS and ⦃θ⦄

s

  • Still one round of messages!

Innovation Centre c 2019 OneSpan Innovation Centre 9

slide-10
SLIDE 10

NEURAL NETWORKS

. . .

Σ

g(l)

j

Activation function x(l)

j

Output x(l−1)

1

θ(l)

j,1

Weights x(l−1)

2

θ(l)

j,2

x(l−1)

dl−1

θ(l)

j,dl−1

Bias θ(l)

j,0

Inputs

Innovation Centre c 2019 OneSpan Innovation Centre 10

slide-11
SLIDE 11

NUMERICAL EXPERIMENTS

  • Implementation (not much optimised)
  • Python
  • Intel i7-4770, 3.4GHz
  • GMP library (power exponentiation)
  • Fixed precision (53 bits)
  • Parameters
  • Public datasets and randomly generated ones
  • Models with 30 to 7994 features
  • Key sizes: 1388 to 2440 bits
  • Message overhead proportional to:
  • Key size
  • Number of features (or number of bits in DGK+)
  • Number of layers (FFNN)

Innovation Centre c 2019 OneSpan Innovation Centre 11

slide-12
SLIDE 12

MESSAGE OVERHEAD

Protocol Protocol step Size (kB) 1 Linear regression Client sends: (core) pkC, ⟦xi⟧, 1 ≤ i ≤ d ℓM + d · 2ℓM ≈ 15 Server sends: t ≈ 2ℓM < 1 SVM classification Client sends (core) t∗, ⟦µi⟧, 0 ≤ i ≤ ℓ − 1 2ℓM + ℓ · 2ℓM ≈ 29 Server sends ⟦h∗

i ⟧, −1 ≤ i ≤ ℓ − 1

(ℓ + 1) · 2ℓM ≈ 30 FFNN sign act. Server sends 2,655 (core) t∗, ⦃µi⦄

s, 0 ≤ i ≤ ℓ − 1

L · d · (ℓ + 1) · 2ℓM (885 per layer) Client sends 2,700 ⟦ˆ y∗⟧, ⦃h∗

i ⦄ s, −1 ≤ i ≤ ℓ − 1

L · d · (ℓ + 2) · 2ℓM (900 per layer)

1Features: d = 30; key-size ℓM = 2048; κ = 95; layers L = 3; Precision P = 53;

Inner-product bound: ℓ = 58

Innovation Centre c 2019 OneSpan Innovation Centre 12

slide-13
SLIDE 13

RESULTS: LINEAR REGRESSION

Private LR: 70 features

1 3 8 8 1 4 7 6 1 5 7 1 6 6 6 1 7 6 6 1 8 7 1 9 7 6 2 8 6 2 2 2 3 1 8 2 4 4 Length of modulus N (bits) 2 4 6 8 10 12 14 16 Average computing time (ms) over 1000 trials Client Server

Private linear regression (core protocol) Dataset: audiology, # features: 70

Private LR: 7994 features

1 3 8 8 1 4 7 6 1 5 7 1 6 6 6 1 7 6 6 1 8 7 1 9 7 6 2 8 6 2 2 2 3 1 8 2 4 4 Length of modulus N (bits) 100 200 300 400 500 600 Average computing time (ms) over 1000 trials Client Server

Private linear regression (core protocol) Dataset: enron, # features: 7994

On Intel i7-4770, 3.4GHz

Innovation Centre c 2019 OneSpan Innovation Centre 13

slide-14
SLIDE 14

RESULTS: SUPPORT VECTOR MACHINE CLASSIFICATION

Private SVM: 70 features

1 3 8 8 1 4 7 6 1 5 7 1 6 6 6 1 7 6 6 1 8 7 1 9 7 6 2 8 6 2 2 2 3 1 8 2 4 4 Length of modulus N (bits) 250 500 750 1000 1250 1500 1750 Average computing time (ms) over 100 trials Client Server

Private SVM classification (core protocol) Dataset: audiology, # features: 70

Private SVM: 7994 features

1 3 8 8 1 4 7 6 1 5 7 1 6 6 6 1 7 6 6 1 8 7 1 9 7 6 2 8 6 2 2 2 3 1 8 2 4 4 Length of modulus N (bits) 250 500 750 1000 1250 1500 1750 Average computing time (ms) over 100 trials Client Server

Private SVM classification (core protocol) Dataset: enron, # features: 7994

On Intel i7-4770, 3.4GHz

DGK+ comparison is the main limiting factor

Innovation Centre c 2019 OneSpan Innovation Centre 14

slide-15
SLIDE 15

RESULTS: NEURAL NETWORKS

Private NNs: 10 features | 3 layers

1 3 8 8 1 4 7 6 1 5 7 1 6 6 6 1 7 6 6 1 8 7 1 9 7 6 2 8 6 2 2 2 3 1 8 2 4 4 Length of modulus N (bits) 100 200 300 400 500 Average computing time (ms) over 100 trials Client Server

simple FFNN with sign activation (heuristic solution) Dataset: random, # features: 10, # layers: 3

Private NNs: 10 features | 3 layers

1 3 8 8 1 4 7 6 1 5 7 1 6 6 6 1 7 6 6 1 8 7 1 9 7 6 2 8 6 2 2 2 3 1 8 2 4 4 Length of modulus N (bits) 10000 20000 30000 40000 50000 Average computing time (ms) over 100 trials Client Server

simple FFNN with sign activation Dataset: random, # features: 10, # layers: 3

On Intel i7-4770, 3.4GHz

DGK+ comparison is the main limiting factor

Innovation Centre c 2019 OneSpan Innovation Centre 15

slide-16
SLIDE 16

COMMENTS/QUESTIONS?

Innovation Centre c 2019 OneSpan Innovation Centre 16