On Robustness of Principal Component Regression Anish Agarwal - - PowerPoint PPT Presentation

1

On Robustness of Principal Component Regression

Anish Agarwal Devavrat Shah, Dennis Shen, Dogyoon Song MIT

2

What is PCR?

1

3

What is PCR?

1

4

What is PCR?

1

Step 1: PCA

What is PCR?

1

Step 1: PCA

(k-components)

5

6

What is PCR?

1

Step 2: Regression

minimize

7

What is PCR?

1

Step 3: Prediction

8

When & Why Use PCR

2

9

Data Science Folklore

2

“IF DATA IS (APPROXIMATELY) LOW-DIMENSIONAL, USE PCR!”

• - LOREM IPSUM
• - An

Anonymous Data ta Scienti tists ts

Whe When n exactly sho should we be usi sing ng PC PCR?

10

Key Questions We Answer

2

Theoretical properties of PCR? Is dimension-reduction only benefit to PCR?

11

Our Theoretical Analysis of PCR helps answer following questions..

How many principal components to pick? How low-rank do covariates need to be? How well does PCR perform on a test data (i.e. generalization properties)?

12

• - LOREM IPSUM

NO!

Is Dimension-Reduction Only Benefit?

13

PCR (as is) works for a wide variety of settings!

2

Missing

? ? ? ?

Mixed valued

1 3. 3.14

Sensitive Noisy

14

• - LOREM IPSUM

We We show PCR R is surprisingly ly robu bust to proble blems ms th that p t plague ue l larg rge-sca scale m modern rn d data tase sets ts

Ma Main in Con

• ntrib

ibut ution ion of

• f this

is Wor

• rk

15

• - LOREM IPSUM

Erro rror-In Vari ariab able Regre ression

(S (Setti etting We e Consider) er)

16

Classical (high-dimensional) Regression

2

17

Error-in-Variable (EIV) Regression

2

? ? ? ?

Representative of modern datasets

18

EIV - Surprising Number of Applications

2

Causal Inference (Synthetic Control) Time Series Analysis Differentially-private Regression Mixed Valued Regression

(noise by design) (measurement noise) (structural noise) (measurement noise)

19

EIV - Surprising Number of Applications

2

Causal Inference (Synthetic Control) Time Series Analysis Differentially-private Regression Mixed Valued Regression

(noise by design) (structural noise) (measurement noise) (measurement noise)

20

• - LOREM IPSUM

Formal R Results

21

Theorem (Informal): Training Error

2

OLS minmax error rate (low-dimensional, noiseless, fully observed covariates) PCR implicitly denoises covariates!

If principal components chosen correctly (" = $)

fraction of observations number of covariates

22

Theorem (Informal): Testing Error

2

If principal components not chosen correctly (" ≠ $)

Test Error Train Error with PCR(")

PCR implicitly de-noises covariates PCR implicitly performs &'-regularization

Choose k that minimizes above

23

When To and Not to Use PCR? – Look at Spectrum

2 Don’t Use PCR! Use PCR!

Magnitude of Singular Values Singular Values (ordered by magnitude)

Case 1 Case 3 Case 2 Case 4

24

Exponential-decaying spectrum is ubiquitous in real-world data

2

GDP Trajectories (Macroeconomics)

25

Avito Ad-Click Dataset (E-Commerce) Exponential-decaying spectrum is ubiquitous in real-world data

2

26

Cricket Trajectories (Sports) Exponential-decaying spectrum is ubiquitous in real-world data

2

27

Surprising Applications of PCR

3

28

Applications of Error-In-Variable Regression

3

Causal Inference (Synthetic Control) Time Series Analysis Differentially-private Regression Mixed Valued Regression

(noise by design) (measurement noise) (structural noise) (measurement noise)

29

Da Data p privacy i is t top-of

• f-mind

mind as s we we inc increasing singly apply ML on n se sensit nsitiv ive use ser data (gene netic ic data, purcha hase se hist history etc.)

30

Standard N Notion o

• f P

Priva vacy i in M ML ε-Differential P Priva vacy

Intuitively, an algorithm is ε-differentially private if ou

• utcom
• me of
• f a

a stati tatisti tical al query ry on a database ca cannot ch change by mo more than ε due to

pr presence/absence of any us user data record

Example of Statistical Query: “Average Income of all users between ages 25 and 30”

31

Ho How w to achie hieve ε-di

differ eren entially priva vacy?

Laplace M Mechanism

database

Laplacian N Noise ⁄ " #

32

Pr Predict ictiv ive Accu ccuracy cy vs.

• s. Pr

Priv ivacy cy Tradeoff ff

Ca Can n we achi hieve good prediction n error and nd still maint ntain n privacy? y? Ye Yes!

33

Pr Predict ictiv ive Accu ccuracy cy vs.

• s. Pr

Priv ivacy cy Tradeoff ff

Ca Can n we achi hieve good prediction n error and nd still maint ntain n privacy? y?

Step 1: Data Owner adds Laplacian Noise Step 2: Analyst Performs PCR

Don Done!

34

Wh What i t is s sample c complexity ty c cost f t for r ε-di differential p privacy?

Prediction Error

Do Does de de-no noising ising st step (PC PCA) break priv ivacy cy? No, PCA only de-noises covariates on average with respect to the - norm

35

Conclusion

4

36

In Inspec pect spec pectrum of yo your cova variate e matrix

Magnitude of Singular Values Singular Values (ordered by magnitude)

Case 1 Case 2

de-noises

Use PCR!

regularizes

37

Po Possib ssible Implica icatio ions ns fo for Modern n ML

Step 1: Dimension Reduction

PCA

Linear Case

Li Linea ear l low-di dimens nsional nal covar ariat ate pre- proc processing has many implicit benefits (e.g. de- noising, regularizing)

Non-Linear Case

Does non-linear covariate pre-processing (e.g. GANs) have similar benefits for unstructured data?

GANs?

38

Co Come Me Meet Us s At Our Post ster Po Post ster #3 #3 – East Exhibition Hall B + C, 5-7pm, Thursday Sh Shameless Plug ug :) PCR for Time Series Analysis: ts tspd pdb.mit. t.edu PCR for Causal Inference: gi github.com/Rom Romcos

• s/SC

SC_de demo